Structure Tax: How Structured Output affects LLMs Performance: what it means for business leaders

A PayPal study finds the accuracy loss blamed on JSON and XML output constraints comes from asking for the answer before the reasoning; put reasoning first and structured output matches or beats free text.

1. What the method is

The Structure Tax paper, from PayPal's artificial intelligence group, examines what happens when a large language model is forced to answer in a structured format such as JSON or XML rather than free text. Earlier studies reported that such constraints lower accuracy and called the loss a structure tax. The authors argue the tax is not inherent to structure but comes from one design choice: whether the schema asks for the answer before the reasoning or after it. A reasoning-first schema lets the model work through the problem before committing to an answer. An answer-first schema makes it commit first and then rationalise. Across five models and four benchmarks, reasoning-first schemas match or beat free-form output, while answer-first schemas can collapse accuracy, especially in smaller models. The paper also shows that the better ordering improves confidence calibration and leaves a measurable trace in the model's internal representations.

2. Why the method was developed

Production systems rarely accept free-form text from a model. Downstream code needs a parseable field for the answer, so an output schema is a commitment teams actually ship, not an optional prompt flourish. Prior work by Tam and colleagues in 2024 and Long and colleagues in 2025 treated each format as a single unit and reported accuracy losses without a causal explanation. The authors noticed that a common production habit places the answer first because it reads naturally to human reviewers, as in a rating followed by a rationale. Since a language model generates tokens strictly left to right, an answer emitted before its reasoning cannot be revised by that reasoning. The paper was written to test whether this ordering, rather than structure itself, explains the reported tax, and to measure how large the effect is across models and task types so practitioners can choose schemas deliberately.

3. Who should care

Anyone running language models behind an interface with a fixed output schema should care, which covers most teams in payments, fintech, customer support automation, document processing and analytics tooling. Engineering leads and platform owners who define JSON or XML contracts between models and application code are the most direct audience, since the fix is a change to field order rather than a new model. Risk, compliance and quality teams that audit model outputs should note the paper's two failure modes and the diagnostics it proposes for each. Teams that use the model's stated confidence to route cases to humans should note that calibration also depends on ordering. Teams deploying small open-weight models in the 4 to 8 billion parameter range are most exposed, because those models suffered the largest collapses. Text-to-SQL and similar code-generation products are the least affected and can keep their current formats.

4. How the method works

The authors compared five output formats on each task: free-form text, JSON and XML with the answer field before the reasoning field, and JSON and XML with reasoning before the answer. All other prompt content was held constant and responses were collected at temperature zero. Four benchmarks were chosen to span a range of intrinsic task structure: Spider for text-to-SQL at the high-structure end, GSM8K arithmetic word problems in the middle, and MMLU multiple-choice knowledge and LogicBench logical reasoning at the low-structure end. Five models were tested: GPT-4o, Gemini Flash 2.0, Claude Sonnet 4.5, Llama 3.1 8B and Gemma 3 4B. Beyond accuracy, the study elicited a verbal confidence score from 1 to 10 to measure calibration. For the two open-weight models it inspected hidden states layer by layer with centred kernel alignment and linear probes to see whether the answer was being computed internally before it was written out.

5. How it was evaluated

Accuracy was measured by exact match on GSM8K, MMLU and LogicBench, and by execution accuracy on Spider, where the generated SQL is run and its result compared with the reference. Each reported cell is the mean of five independent runs with the standard deviation, over 1,319 GSM8K problems, 500 LogicBench items, 1,531 MMLU questions and 1,030 Spider queries. Calibration used expected calibration error, an overconfidence rate and the Spearman correlation between stated confidence and correctness. A robustness check reran GSM8K on Llama 3.1 8B and GPT-4o with three variants of instruction phrasing, schema casing and whitespace per condition. Reasoning length in words was recorded across all 80 model, dataset and format combinations. The representational analysis was limited to Llama 3.1 8B and Gemma 3 4B. The authors report zero schema violations in structured runs, so parsing failures did not drive the accuracy differences.

6. How it performed

Reasoning-first JSON or XML took the best cell in 13 of 20 model and dataset pairs. The sharpest collapse was on GSM8K arithmetic, where answer-first XML dropped Llama 3.1 8B and Gemma 3 4B to about 11 percent accuracy against 83 to 87 percent with reasoning first. GPT-4o fell from 89 percent free-form to 55 percent under answer-first JSON and rose to 96 percent reasoning-first. On Spider, free-form stayed competitive for three of five models and ordering barely mattered. Within reasoning-first schemas, XML beat JSON by up to 12 points. Reasoning-first XML also lowered calibration error relative to free-form on LogicBench for every model and on MMLU for four of five, though per-example confidence remained only weakly predictive of correctness. The ordering gap survived prompt perturbation, never shrinking below 37 points on GSM8K. The authors limit their claims to prompting rather than fine-tuning, two orderings only, and four coarse task types. (Source: arXiv 2610.12056, 2026)