Can LLMs Ignore What They Know? Measuring Knowledge Leakage in Persona-Based User Simulation: what it means for business leaders

An LLM persona told it has never heard of a product will say so, yet it still rates the real product above a fabricated look-alike, which skews synthetic surveys and simulated-user tests.

1. What the method is

The paper names a failure it calls knowledge leakage: facts that a simulated user could not know end up shaping that user's simulated answer. It also gives a way to measure it, the impostor-paired contrast. Each real survey item is an obscure but verifiable term, such as psyllium husk. It is paired with a fabricated twin that looks like it, such as Psyllyer humb, and the twin has been screened so that none of the tested models claims to recognise it. The same low-knowledge persona rates both on a seven-point scale, and the contrast is the real rating minus the twin rating. The twin carries the item's surface form, its category and the persona's response style, but no stored facts. So a persistent gap means the persona is using knowledge it was told it does not have. The method works on subjective ratings that have no correct answer, and it needs only black-box API access.

2. Why the method was developed

Using one model to play a thousand survey respondents has become standard practice in market research and social science. Most validation so far compares simulated answers with human answers in aggregate. That can show a mismatch but cannot say where it comes from. The obvious check, asking the persona whether it knows the item, is circular: the answer comes from the same simulation being audited, and a persona that leaks knowledge into its ratings can also produce compliant self-reports. Earlier role-play studies measured knowledge-boundary violations mostly on questions with verifiable answers. Ratings of products, brands or organisations have no right answer to expose contamination. The problem also reaches beyond surveys. A simulated user in an agent benchmark who quietly knows the hidden task solution may cooperate in ways real users would not and inflate agent scores. Prior work found that the choice of simulator alone shifts measured agent success by up to 9 percentage points.

3. Who should care

Product and marketing teams that use synthetic panels for concept tests, naming, brand awareness or purchase intent should care most. The low-knowledge persona this paper tests is exactly the respondent a launch study needs: someone who has never heard of us. ML evaluation teams that test chatbots or agents against LLM-simulated users should care because leaked knowledge can make simulated users unrealistically helpful and flatter the system under test. That applies directly to our own simulated-user A/B testing work, where personas are supposed to know only what their description says. Procurement and research leads who buy synthetic-respondent services from vendors now have a concrete question to ask: do you seed questionnaires with fabricated controls and report the gap for each question type? Data science leads who read pooled validation results should note that a pooled check can pass while the bias is present.

4. How the method works

For each real item, a model from outside the tested families and an algorithmic nonword generator produced twin candidates matched in morphology and implied category. The candidates were screened on character n-grams, token counts and surprisal, then run through an adversarial realness probe. Any candidate a study model claimed to recognise was discarded. Of 1,158 candidates, 697 were discarded and 461 survived, and 178 were used, one per real item. The persona's ignorance was set up three ways: a familiarity question asked before the rating, the same question asked after it, or a sentence in the prompt stating the person has never heard of that specific item. Items were coded as favourable or unfavourable from reference sources. Half the sum of the two contrasts estimates a familiarity effect that is independent of the facts. Half the difference estimates an effect that follows the facts. That split assumes the two content effects are equal in size and opposite in sign.

5. How it was evaluated

The rating experiment ran 145,560 trials over 404 items: 178 real items (88 favourable, 90 unfavourable), their 178 twins and 48 fillers. The items covered six domains: supplements, skincare, food additives, household chemicals, brands and civic organisations. It used five commercial endpoints from two providers: gemini-3.5-flash, gemini-3.5-flash-lite and three gpt-5.6 variants, all at temperature 1.0 with tools disabled. Each domain had three low-knowledge profiles, one expert profile and a no-persona control. Scale direction was randomised and two wordings were used. Confidence intervals come from 10,000 bootstrap resamples clustered on the real item. Only 0.31 percent of responses failed to parse. A separate probe of 768 calls asked personas on two endpoints whether they recognised 24 real items. No human respondents were measured at any point, so every result shows divergence from the researchers' specification, not from a real population.

6. How it performed

Self-report looked clean. Told they had never heard of an item, the personas claimed recognition in 0 of 96 questions at each endpoint. The ratings told a different story. Favourable real items were rated above their twins under the explicit-ignorance sentence, with intervals excluding zero at all five endpoints and for every profile; estimates varied twofold across endpoints. 72 of 88 favourable items beat their twin, and the effect held after dropping the ten largest. The pooled unfavourable contrast sat near zero, but that average hid two opposite effects: it was negative where the scale asked about hazards and positive where it asked about purchase intent or organisations. This split was exploratory. The rendered text we read lost the exact rating-point magnitudes, so we quote none. Limits: no human baseline, the screening used the same study models, lexical differences remain, and only commercial APIs were tested. Our call: pilot it, by putting screened twins into our own simulated-user evaluations. (Source: arXiv 2610.39939, 2026)