Presentation Sensitivity in Typed Decision Models for Subjective Text Evaluation: what it means for business leaders

A 702,000-judgment audit shows that three low-cost typed decision models repeat themselves reliably yet change verdicts when order, endorsements, framing or rating rubrics change, so stable outputs do not prove a trustworthy evaluator.

1. What the method is

This paper is an audit, not a new model. Agrawal and Mamidi measure how much three typed decision models, Jev from TypeSafe, Clef Flash from Cloudflare and Perplexity Decider v1.1, change their judgments when only the presentation of a request changes and the evaluated text stays identical. Typed decision models return choices, scores and probabilities directly, without generated text or an explanation. The authors vary candidate position, identifiers, claimed authorship, reader and prior-judge endorsements, evaluator directives, selection versus rejection framing, a no-preference option and rating rubrics, then collect 702,000 judgments on humor, helpfulness and summary quality. The advance is the controlled design, which separates these effects from each other and from ordinary call-to-call noise, along with a public repository of scripts, request specifications and aggregate results that we could point at other decision models.

2. Why the method was developed

Typed decision models promise fast, inexpensive subjective judgment at scale. TypeSafe's own workflow benchmarks claim large cost and speed advantages over the language-model configurations it compared, and evaluation tools already document Jev for response grading. Those are vendor figures, not findings of this study. Earlier research established that conventional LLM judges are swayed by response order, authority cues and rating formats, and that agreement with people depends on the task and on whose judgment counts. That work centered on conventional judges and left open whether this newer model class, with its decision-specific architecture and no chain of thought to inspect, behaves the same way. With no explanation to read, behavior under controlled changes is the main evidence available about how a verdict was reached. The authors set out to supply that evidence as these models enter evaluation pipelines.

3. Who should care

ML platform and evaluation engineers should care first, because any release gate, regression suite or model comparison built on these judges inherits the effects measured here, and fixed score thresholds are the most exposed. Product leads who read aggregate quality dashboards should know that a rubric rewrite can move the number with no change in the underlying product. Procurement and vendor managers get a concrete test to demand before buying cheap scoring, since the three models fail differently and a pass on one tells us little about another. Data teams that use automated labels for training or ranking should check that order and metadata are not leaking into requests. Security and trust staff should note that endorsement text placed near a candidate moves verdicts, which matters wherever outside parties can influence what the judge sees.

4. How the method works

Each model is called through OpenRouter's Decisions API with the same request construction. A choice request gives two candidate texts as options and the task context as shared state; a score request gives one text plus an ordered list of verbal anchors. A factorial design swaps labels, submitted positions or both, which isolates position bias from identifier bias. Cue experiments attach separate metadata to unchanged text, such as 80% versus 20% of previous readers preferring an option, then reverse the assignment and add a not-provided control, so the effect of the cue's direction is split from the effect of the added format. Rejection framing asks for the worse option and takes the complement. Rubric tests reverse the anchors or use three or seven levels. Four unchanged repeats beyond the baseline set the noise floor. That makes 28 choice and 24 score runs per dataset and model.

5. How it was evaluated

The study uses six public datasets, a choice set and a score set for each task: UKP Humor and Humicroedit for humor, HelpSteer2-Preference and HelpSteer2 for helpfulness, and OpenAI summary comparisons and SummEval for summaries. Each contributes 1,500 multiply annotated items, 750 with clearer human agreement and 750 more disputed, selected with a fixed seed. Human-preferred candidates are balanced across positions, and human votes are withheld from requests. References are aggregates such as majority votes or mean ratings. There is no competing judge in the experiment: each intervention is compared with its matched control on the same items, and conventional LLM judges appear only through published work. Confidence intervals account for reused content, and Benjamini–Yekutieli correction covers 369 primary tests and a separate secondary family of 1,656. The copy we reviewed cuts off in the appendices, so this page rests on the main body.

6. How it performed

Unchanged calls flipped 1.70 to 3.20% of choices for Jev and Clef and at most 0.08% for Perplexity, with baseline human agreement of 61.85 to 78.98%. Presentation moved far more. Reversing reader endorsements changed Perplexity's selected text on 40.40% of humor pairs and 27.13 to 34.20% on the other tasks, despite its near-perfect repeatability. Jev's second-position advantage reached 6.11 points on summaries. Human attribution moved Clef's summary preferences by 32.15 points, and rejection framing cut its summary agreement by 23.36 points. Rubric changes shifted Jev's scores by up to 7.63 points while rank correlations stayed at 0.963 to 0.996, so rankings hold up better than thresholds. The authors caution that results cover three English tasks and this exact wording, with no matched test against conventional judges. Our call: pilot these models for rank-based comparisons only, behind our own audit of order, cues, framing and rubrics, and hold off on fixed score thresholds. (Source: arXiv 2611.06384, 2026)