‘is it different when you chat with it?’

links

benchmark page
download results as json

i always wanted a benchmark that simply measures if the model has an unusual or lateral character, and is willing to dig into the latent space when writing. that’s why i created this benchmark.

take this bench with a big grain of salt, as it is a character-vibe check, not a measure of instruction-following intelligence. even though i consider it a success in terms of my initial goal with some additions (like it may measure generalization beyond mainstream topics and idea-generation (red teaming)), it is far from perfect. for those who discover that its vibe aligns well with their experience, chatting, this benchmark will be a shortcut. for those that don’t, either flip it (use lowest models for eli5 etc.) or discard it.

thoughts

non-overlap bench punishes overlaps with ‘common answer keywords’ across n-grams (word sequences). the goal is to intuitively measure how ‘different’ the llm model behaves compared to the calibrated crowd.

for example, some models love to use word-sequences like:

“you should,you are describing,should seek,should always,I totally,you are, spend more, absolutely right,totally get you,utilize some,kinda,consult an expert,everything,I need to empathize,the intricate,complex,a multifaceted,nuanced approach,profound,guide you,practice mindfulness”

in easy terms, we search for and penalize this on a larger scale; word sequence checkers mark the path of how models usually respond. when a model is tested and doesn’t cross the path, the benchmark pets them and gives them a cookie. models with a high ‘median non-overlap_score’ successfully suppressed high-probability associations (the common path), didn’t attempt casual conversations, and found isomorphisms in obscure domains.

single-word overlaps receive low penalties because they measure lexical usage exclusively. many n1-n2 sequences are whitelisted (like “it, the,because,to the,this is”). longer word sequences like “it is important to” receive higher penalties because they measure how cheaply the model uses words (copy/pasting sentences). for an llm, excessive familiarity with a common subject creates a liability in non-overlapping scenarios, where vast training data traps the model in the gravity of consensus thinking.

research

why associations?

i found ’list association/isomorphism’ requests to be the most consistent, as it is always easy to point out the most obvious association and ignore the ones in distant domains. with my initial intention of measuring “novelty”, i seek to incentivize novelty at the very heart of the request.

what kind of statements (within requests) make an impact?

measurably, statements that produce varying non-overlap scores across models. cliché traps for associations, standard curricula, textbook concepts, and tropes involve cultural ubiquity that models with overfit historical ‘crystallized intelligence’ confidently assign. doing so is a sign of lacking effort to explore new meanings to existing concepts. a request-trap has a few characteristics:

uses implicit bridges to a famous law or discovery (cliché).
keeps the concept broad with few specific constraints (abstract).
is short and punchy (laconic) with few individual details to pick up.

even though hard-to-escape specific-domain association requests were tried, they didn’t create a characteristic profile across models per request and flatlined in scores. this is probably due to a higher association space when really specific or novel requests are made. models have different attention and might pick a smaller portion to extrapolate. this resulted in me performing a big reduction/filtering in the existing dataset.

how many requests are sent?

tldr: check the benchmark page description. analyzing the “character” or “style” of a model across all its tokens yields more data per response than benchmarks, which reward responses on binary criteria. thus, response quality matters over quantity. generally, i use a minimum of 20. stdev and word-count will tell the bigger story of how viable this is.

how does response length impact scores?

the benchmark was designed with word length in mind. specifically, we take the full word length of responses and calculate the proportion of overlaps. this is why median n penalty and total n overlap are not directly correlated. a higher overlap is tolerated when a higher word count is used.

even if the trend isn’t strong, the linear regression shows that models choosing to respond with fewer words get a slight edge. this is probably due to inevitable n4 overlaps with response length (higher long n-gram exposure). i still encourage you to use the ’total words’ scale and see for yourself.

problems surfaced

models responding with gibberish, mathematical special chars, borrowed unfit terms/hallucinated numbers, simply loosing focus, have no distinction in scores from focussed but lateral ones. the n-overlap aims to add nuance to this but still fails overall. in fact, low-scoring models can act more grounded in complex code-bases (eli5 descriptions).

an improved version would track pareto focus-on-topic vs novelty-of-ideas, potentially measuring model-initiative in constrained environments. dynamic riddles could power this; like a requirement to fulfill a set of conditions, penalizing common pathways to get there.

technical

evaluation workflow:

through iterative testing, requests are designed, serving as prompts for the dataset. especially, abstract and universal domains were selected.
common answer keywords are derived from ~70 tiny or cheap models over multiple runs, creating a vast dataset on what word sequences not to use - this aims to model the crowd-consensus-response characteristics. it requires frequent updates because old tiny models often depreciate, requiring new ones.
responses are scanned for frequent overlaps across n-grams (1-4). the benchmark exclusively penalizes the deployment of such terminology, thereby incentivizing any divergent output (even incorrect answers) - to be fair, inputs are inherently abstract and open-ended, reducing any need for ‘factual accuracy’.

example queries (may still be included):

exclusively list 10 isomorphisms to this conceptual/thematic statement: ‘a specific frequency will shatter glass, others just pass by.’ - response-length budget is 500–600 words.
exclusively list 10 isomorphisms to this conceptual/thematic statement: ’the ultimate victory is not solving the unsolvable problem, but finding absolute joy and purpose in the futile, repetitive act of trying.’ - response-length budget is 500–600 words.
exclusively list 10 isomorphisms to this conceptual/thematic statement: ‘hft liquidity vanishes under stress, depth is a calm-only illusion.’ - response-length budget is 500–600 words.

all queries follow this pattern:

’list 10 associations/isomorphisms to this statement ‘statement’ + response length soft rule.’

models are never instructed to “be creative” or unconventional. the purpose is to evaluate how they respond naturally. this bench doesn’t rely on clear objectives. when associations are lateral or ’novel’, they won’t overlap, making the model score high (few penalties subtracted from the score).

future

ideas and potential features: embedding models or sentence-similarity checkers, comparing to low-quality outputs like now, web-results, or n-back sentences to measure pace. static n-gram overlaps always have gaps impacting accuracy in the evolving field of transformers. i suggest keeping the core-purpose of measuring deviance, just with more adaptive and dynamic tools.

since its increasingly hard to keep up with model releases (& efforts), benchmark large-enough samples, paying for all cost-to-run, and standard baselines change with time, this project won’t be a continuous effort but a sequential sample. the results could have interesting patterns that tell us more about model-architectures and training methods. for collabo requests, catch me up.