Judged in Company: In-Context Batching Makes LLM Verdicts Depend on Other Items: what it means for business leaders

Grading many items in one LLM prompt changes about one decision in seven on ambiguous tasks, while accuracy on a labelled sample barely moves, so our usual validation would never notice.

1. What the method is

The paper is a measurement study, not a new model: it tests whether an LLM's verdict on one item changes when that item is graded in a prompt alongside other items. The authors call the assumption it tests separability, meaning the verdict depends only on the item, not on which other items share the prompt, their order, or the item's slot. They grade the same 2,000 items per task alone and in random batches of 25 under five open-weight models of 4 to 9 billion parameters, with four 14 to 32 billion models in a follow-up. They show that the rate at which decisions flip between two random batchings decomposes exactly into components for neighbour order, slot and batch membership, and under a threshold model they derive that flips grow with the square root of the variance the context adds, which is why small score shifts become large decision changes.

2. Why the method was developed

Batching many items into one prompt is the standard way to cut the cost of LLM labelling, grading and moderation, because the instructions are paid for once instead of per item. Vendors recommend it for non-real-time moderation, and a recent annotation study found batches of 25 to 100 items lose under 2 points of accuracy while saving over 80 percent of token cost. That evidence rests on aggregate accuracy, which the authors argue is the wrong test: a decoder-only model writing the verdict for the twelfth item has already read the other 24 items and its own eleven earlier verdicts, and nothing in the architecture prevents those from leaking in. Shifts that push some items up and others down can leave accuracy almost unchanged while many individual decisions differ. Earlier work on position bias and voting over permutations measured accuracy, not whether each decision changed.

3. Who should care

Anyone in our company who runs or buys an LLM pipeline that returns one verdict per item is affected: trust and safety teams moderating comments, product teams grading free-text answers or support tickets, data teams producing labels for training sets, and the ML platform group that chose batching to control inference spend. Engineering leads need to know that the usual validation, accuracy on a labelled sample, will approve a batched pipeline whose decisions are partly arbitrary. Legal and compliance should care where a verdict touches a person, since a comment flagged as harassment or an answer marked wrong can depend on which other items happened to share the prompt. Finance should expect a conversation about inference cost, because the fix is to grade ambiguous items one per prompt with prompt caching, which is cheaper than it sounds but not free. Vendors we evaluate should be asked for the agreement numbers this paper proposes.

4. How the method works

Each item is graded alone, in three independent random partitions into batches of 25, and in derived conditions that each change a known subset of three factors: the order of an item's neighbours, its slot, and which items are its neighbours. Comparing flip rates across these conditions yields the variance decomposition of Hoeffding and Sobol. Baselines measured in the same regime, renamed item keys, a reworded instruction and the same prompts generated one at a time, bound what trivial changes and numerical noise alone would cause. A threshold model, where decisions threshold a latent score and the scores in two contexts are bivariate normal, turns flip rates into tetrachoric correlations, giving a scale independent of how many items sit near the threshold. They also cut a generation just before the target's digit and edit either the earlier verdicts or the neighbours' texts, reading the next-token distribution to see through which channel the batch acts.

5. How it was evaluated

The authors used three tasks of 2,000 items each: harm in social-media comments from the Measuring Hate Speech corpus on a five-level scale with the decision at level 2 or above, short-answer grading from SciEntsBank as incorrect, partial or correct, and Yelp star ratings with the decision at four stars or more. Main models were Qwen3-4B, Qwen3-8B, Gemma-4-E4B, Granite-3.3-8B and GLM-4-9B in bf16 with greedy decoding on one RTX 5090 each; follow-ups added Qwen3-14B, Qwen3-32B, Gemma-4-31B and GLM-4-32B on H200 GPUs. Prompts ask for a JSON object mapping each item key to a digit. Three pass or fail gates on generality, membership and mechanism were fixed before the runs. Intervals are bootstraps over batches or items. Interventions used 200 targets per task in slots 12 and 24. Voting experiments graded every batch of two partitions in nine orders and added twelve further partitions to give fifteen independent batches per item.

6. How it performed

On harm and grading a new random partition changed 11 to 22 percent of decisions for the 4 to 9B models and 6 to 15 percent for the 14 to 32B ones, 4 to 13 points above the largest trivial-change baseline, while AUROC against human labels moved by only a few hundredths. Order and slot carried most of the effect, but the membership main effect, which voting over permutations cannot remove, was 0.11 to 0.22 of the latent dissimilarity, and majority votes over nine permutations still left 52 to 60 percent of flips at nine times the cost. The majority of fifteen batches disagreed with single-item grading on 8.4 to 17.8 percent of decisions. On star ratings only 2.9 to 3.9 percent changed. The authors tested only open-weight models, English tasks and one prompt template; proprietary models may differ. Our call: pilot, by measuring batched against single-item agreement on our own ambiguous tasks before trusting any batched verdict pipeline. (Source: arXiv 2610.40241, 2026)