v3.1 | drafting an evaluation environment

when is there potential?

this is a simple but hard llm benchmark that tests how generic the model responds.

preparation:

1. a question, request, or statement is parsed to an ensemble of 25-30 small, dumb models. (1b-70b)
2. frequent keywords are extracted as common answer keywords.
3. these keywords (300-1500) are attached as ‘common answer keywords’ to the initial question, request, or statement.
4. global generic common answer keywords are defined. (~800)
5. global whitelist keywords are defined (like ‘it’, ’the’, ‘or’)

execution:

1. a python script sends all questions to the openrouter api (0 temp, no system prompt; if system prompt, then a respond-length instruction) and extracts alphabetic words from received responses (above 1 character words).
2. then scans for overlaps and gives scores per question, request, or statement using this formula: response_word_count / response_overlap_with_keywords

reasoning:

by comparing how unusual the model responds in comparison to dumb models, novel responding styles can be evaluated. this doesn’t measure the intelligence or factual correctness and is definitely not fool-proof. in fact, it could even reward scores if the model suddenly responds in spanish. thats why i manually re-read the chat log with unusual syntax highlighting (inefficient method), to validate good scores (yes, some models hallucinate even at 0 temp). after all, if it uses a spanish word in an english response, it might be more semantically fitting, brings more variance, and lets me learn something new, but if the entire response is unreadable, the result is dismissed.

this post serves as a general blueprint draft reference. details in linked benchmarks may vary.