a post to donate types of prompts that could be used as a benchmark.

criteria guideline:

  • problems that llm had struggled to solve. ( and somewhat desirable )
  • valuable usecases
  • centric to various other problems ( with value )
  • located in a domain that underperforms on all tasks related to it

* soft push for actual personally useful problems. not toy demonstrations/tests . * no criticizing/debating . this is write-only
* soft focus for smaller models ( rationale being: no really hard problems that are cutting-edge math,science,engineering ) . eg ok to be using large models but for not cutting-edge problems that a smaller model definitely could never handle.
* one post per user
* vague description of problem/prompts is ok.

  • leanleft@lemmy.mlOPEnglish
    1·
    3 days ago
    • homeless knowledge ; ( as you would expect… poor knowledge across the board )
    • perl5 coding ; ( not as good as python ).
    • usa knowledge and usa society ; ( you would expect alot better )
    • simple geographic trip planning with constraints andor weighted decision making, that involves graph
    • ptsd response ; ( failed )
    • notetaking ; ( not as good as you would expect. also gives cheesy obvious answers )
    • android OS
    • firefox about:config
    • OSM
    • relationships ( not bad but overconfident and not a careful thinker. prompting is sensitive. also sometimes cheesy obvious answers)
    • meaning of life
    • designing new things with reasoning ( not just reproducing a synthesis or variation ) ; feels similar to multishot overfitting problem.
    • problem solving ; ( not as good as you would expect )
    • certain security problems ; ( outperformed by human commonsense )
    • telecom products/companys ; ( blatantly wrong or censored info )
    • BSD
    • data ; ( regarding special case: non-enterprise )
    • sexy chat ; ( models that *i* use… failed. cheesy answers. refused to get kinky. im sure theres a couple out there. )
    • crime scene investigation/forensic ; ( not an expert. however i detected multiple missing explainations which i think are fairly plausible and worth mentioning )