Forty Questions: Identifying a Language Model from One-Word Answers: what it means for business leaders

Forty one-word answers to everyday questions pick the exact model behind an API out of 94 candidates, and an audit costing $7.20 found a provider serving Gemma 4 under the Llama 4 Maverick name.

1. What the method is

The paper, by Tapan Parikh of Cornell Tech, shows that you can tell which model sits behind an API from its one-word answers to everyday prompts such as "Name a sound" or "Name a poet." Each candidate model has a profile: how often it gives each answer to each of 96 questions, built from eight samples per question at temperature 1. New answers are credited to the profiled model most likely to have produced them, using a naive Bayes score with a small pseudocount. Nothing is trained beyond those counts, so adding a model to the panel means sampling it once. On a panel of 94 models, 40 answers name the exact model 92.5% of the time, against 1.1% by chance. Every decision traces back to the specific answers and to the models that give them, so a person can check it by reading.

2. Why the method was developed

When we buy inference through an API, we see a model name, not the weights or the serving stack. A provider can quantize the model, add a default system prompt, switch reasoning off or substitute a different model, and none of these changes the label on the invoice. Earlier black-box audits fell short in different ways. Some relied on trained classifiers over requests for random digits and bit strings (IRIS, FLIPS, LLMmap). Others used distribution tests such as Model Equality Testing, which say that an endpoint differs but not what it is, and which falsely reject honest models when a hidden prompt has been added. Single-token fingerprinting with Jensen–Shannon distance assigned the right family only 59.5% of the time. The author wanted identification that is exact, cheap and readable by a person, built on a public census of model defaults.

3. Who should care

Procurement and vendor management should care first. The audit found one provider serving Google's Gemma 4 under the name Llama 4 Maverick, and prices for a single model vary up to thirtyfold across OpenRouter providers. Platform and ML engineering teams that route traffic through gateways can run this as a recurring check that the model we tested is still the model we receive. A change would show up within a few dozen calls. Security, compliance and model-risk owners get an evidence trail for vendor claims, including whether a vendor's "own" model answers like another lab's. Finance gets a related signal: billed input tokens revealed hidden prefixes, such as 1,247 tokens billed to Grok 4.7 for a one-line question. Teams that ship models behind an API should assume their customers can now run this check on them.

4. How the method works

Each of the 96 questions asks for one word in a category such as color, bird, villain or smell. Replies are normalized first: reasoning blocks and lead-ins are stripped, case and accents are folded, and spelling variants and plurals are merged using a hand-checked map, so "A common color is blue." becomes "blue". A model's profile is the count of each answer per question. A test set is scored by adding up, across its answers, the log of each answer's smoothed frequency in each candidate's profile, with a pseudocount of 0.1. The highest score wins. The signal comes from departures from the consensus that repeat: a model that says "petrichor" where most say "lavender" says it again. At small budgets, greedy question selection adds accuracy. For audits, billed input tokens are also compared with peers that share a tokenizer, which exposes prefixes the caller did not send.

5. How it was evaluated

The panel is 94 models from 27 families, each with at least eight answers on at least 90 of the questions. All were queried through OpenRouter's default routing between July and October 2026, with no system prompt and at temperature 1. Identification was measured with four-fold cross-validation over the eight samples per question, so no answer is scored against itself, and random question subsets were drawn ten times per size. The likelihood scorer was compared with two simpler ones: Jensen–Shannon distance and favourite-answer matching. Robustness tests covered ten models queried on DeepInfra under blinded code names, fourteen models under two system prompts, and stability over several weeks. The validation was an audit of 96 pinned endpoints serving 13 open models on OpenRouter, each asked all questions four times. It used peer permutation tests, billed-token checks, reference profiles rebuilt from pinned peers, and follow-up runs with reasoning off plus a probe asking each model to identify itself.

6. How it performed

Forty answers identify the exact model 92.5% of the time, and all 96 questions asked twice reach 98.7%. Ten selected questions asked twice reach 81%, against 67.5% for ten random ones. At forty answers the likelihood scorer reached 92%, against 65% for Jensen–Shannon distance and 76% for favourite answers. All ten blinded DeepInfra models were identified. A system prompt cost 6 to 16 points. In the audit, 92 of 96 endpoints matched their claim. Of the other four, one was consistent with Kimi K2.5 served with reasoning off, one was Gemma 4 sold as Llama 4 Maverick (confirmed by its own self-description and its billed token counts), and two remain unresolved. The whole audit cost $7.20. The author lists the limits: it only chooses among models already profiled, the profiles were built through mixed routing, billing is only a lower bound on hidden context, everything ran at temperature 1, and no test was run against a provider gaming the public questions. Our call: pilot it as a recurring check on our inference vendors. (Source: arXiv 2610.28281, 2026)