What Coding Agents Leave Unsaid: Measuring Task-State Gaps Between Execution and Reports: what it means for business leaders

A Tencent measurement study finds that readers of a coding agent's progress reports recover only about half the task state that the full execution record supports, and the gap barely shrinks after the fix is verified.

1. What the method is

CTXGAP is a measurement instrument, not a new agent. It quantifies how much of a coding agent's task state, meaning failed attempts, workarounds, decisions, environment changes and unresolved risks, a reader can recover from the agent's accumulated user-facing reports compared with its full recorded execution context. The authors, a Tencent team, extract factual propositions from each recorded trajectory, turn them into questions with reference answers, and have the same model answer each question twice: once reading only the reports and once reading everything the agent saw and did. On 37 SWE-bench Verified tasks and 4,799 valid questions, report-only readers recovered 52.0 percent of the state while full-context readers recovered 92.7 percent, a 40.7-point gap. The benchmark and code are public, so the method can be applied to our own agent transcripts rather than only to theirs.

2. Why the method was developed

We increasingly hand long-running coding work to agents and judge progress from what they tell us. A supervisor cannot investigate a failure they were never told occurred, and the report is often the only channel. Existing benchmarks such as SWE-bench score whether the issue got fixed; they say nothing about whether the risks found on the way reached anyone. Earlier work on agent self-reports counted whether actions were mentioned or whether logs could be reconstructed, but the authors show that mention is not enough: a report can say a fake database client was used for testing without saying which scripts did what. Their motivating case is a Django trajectory where the agent discovered that a password change followed by a request from another device could raise an HTTP 500 for certain custom user models, and the reports never conveyed the trigger.

3. Who should care

Engineering leaders who approve agent-authored changes should care most, because the number says that roughly half of what an agent knows about its own work does not reach the person signing off, even after a verified fix. Platform and developer-productivity teams that build or configure agent harnesses get a concrete evaluation target for report design, separate from pass rates. Security and code-review functions should note that unreported risks and aborted verification steps are among the categories most often lost. Procurement and vendor-evaluation roles can ask agent tool vendors how their reports score on this dimension rather than relying on completion rates alone. Teams that chain agents, where one model reviews another's summary, are exposed in the same way, since the gap persists when the reader is a model rather than a person.

4. How the method works

Each task runs through four checkpoints with fixed instructions: investigate without editing, implement a fix, run tests and report, then assess remaining risk. At every checkpoint the pipeline records the trajectory prefix and a workspace snapshot. A model extracts atomic task-state propositions from the execution record, each tagged with a category such as failure, progress, risk or decision, and an importance level; only levels two and three are sampled. For each sampled atom it generates a question and a reference answer. An observer model answers the question three times from the report view and three times from the full-context view, and a separate judge awards full, partial or no credit. Probes are weighted so category sampling does not distort the mean, and confidence intervals bootstrap over issues. A reading ladder adds terminal scrollback as an intermediate view, and the agent itself answers from a forked session as a check on what it actually knew.

5. How it was evaluated

The headline cohort is 37 SWE-bench Verified Python tasks, 148 checkpoints and 4,799 valid probes, with HY4-preview as the executing model and also the model behind extraction, probing, answering and most grading. The authors state the cohort was not randomly drawn and overrepresents easy issues; an expanded 76-task set is reported as a sensitivity check. Hidden tests label 30 of the 37 tasks as fixed at the final checkpoint. Validity checks include two human annotators classifying 195 unrecovered atoms as unmentioned, mentioned but insufficient, or sufficient; a 300-probe reference-paraphrase audit; a dual-rater scoring audit; and HY4 and DeepSeek crossed as question writers and answerers. Robustness runs repeat the measurement across the Claude Code, Pi and DSH harnesses on 12 shared issues, with DeepSeek-v4-flash as an independent executor, and with a fully separate DeepSeek extraction-to-grading pipeline. Reporting interventions compare original reports against context-informed rewrites at matched total length.

6. How it performed

Agents answered 93.6 percent of questions correctly when asked directly, full-context readers recovered 92.7 percent, and report-only readers 52.0 percent, a 40.7-point gap. At the final checkpoint of the 30 verified fixes the gap was still 34.8 points and positive in 28 of 30 tasks. Full terminal scrollback lifted recovery to 73.6 percent, closing about half the gap. Human annotators attributed 55 to 59 percent of the lost mass to states that were mentioned without enough detail, and adding a bare mention produced no measurable gain. Rewrites informed by the full context improved recovery by about eight points at matched length, with no advantage for targeted over natural rewriting. Gaps of 39 to 49 points recurred across harnesses, and the independent DeepSeek chain measured 39.8. The authors caution that probes are model-generated, the cohort is non-random, and no human supervision study was run. We should pilot CTXGAP on our own agent transcripts before trusting report-based sign-off. (Source: arXiv 2610.20003, 2026)