Where Is the Truth When AI Agents Disagree? Dependence-Aware Belief Propagation for Unsupervised Truth Discovery (Extended Version): what it means for business leaders

When LLM agents built on the same base model agree, BP-TD counts them as about one vote, which beats majority vote without labels on some agent pools but backfires when the largest group is the strongest.

1. What the method is

BP-TD is a method for inferring the correct answer, and each agent's reliability, when several LLM agents disagree and no labelled answers exist. It runs mean-field variational inference on a one-coin Dawid-Skene model, the standard crowdsourcing model in which each source gives the truth with some probability and otherwise picks a wrong option uniformly. It adds one thing: a per-agent decorrelation weight that makes agents sharing a base model count together as about one vote. The groups come either from provenance, meaning which base model sits behind each agent, or from raw answer co-occurrence. The first iteration is exactly a weighted plurality vote, and later iterations add reliability estimates. The advance is the weighting layer and a map of when it helps, not the inference machinery, which is established.

2. Why the method was developed

Many deployed agents are variants of one model: prompt personas, self-consistency samples or retrieval wrappers. Their errors are correlated, so their agreement is weak evidence of correctness. Classical truth discovery, two decades of work on conflicting web sources and crowd workers, assumes sources are conditionally independent given the truth and scores reliability by agreement with the consensus. On a correlated pool it rewards the agreeing bloc, ranks it as most reliable and can fall below a plain majority vote. The paper shows the same implementations work on independent agents, so the dependence is the cause rather than the estimators. Copy detection based on shared false answers does not fix this either: when the dependent bloc is the majority, it defines the current truth estimate, appears to share no errors and escapes the discount.

3. Who should care

Engineers building multi-agent pipelines, debates or ensembles are the first audience, because they must commit to one answer when agents conflict and usually default to voting. Evaluation and quality leads gain a label-free way to rank agents, useful for audits when ground truth is expensive or unavailable. Those who buy agents should take a simpler lesson: several products agreeing is not independent confirmation if they share a base model, so we should ask vendors what sits underneath. Budget owners should note the finding that cuts against more aggregation: a small labelling effort used to choose one agent outperformed every aggregator tested on three of the four benchmarks. Teams running a single strong model, or with labels readily available, have little reason to adopt this.

4. How the method works

The method alternates two closed-form updates over a graph linking agents, questions and candidate answers. The truth update is a weighted vote: each agent adds a reliability-based score to the answer it chose, scaled by its decorrelation weight, and a smaller score to the alternatives. The reliability update is Beta accounting: each agent accumulates the current belief placed on the answers it gave. Family weights divide a vote by the size of the base-model group and need only provenance. Co-occurrence weights discount an agent by how many others agree with it above a threshold, which can be fixed in advance or set by the label-free Otsu rule, and never consult the truth estimate. The weights are a heuristic tempering of the likelihood, not a generative model of dependence. Per iteration, cost is linear in answers and quadratic in agents; one pass runs in under a second on CPU.

5. How it was evaluated

The pool has 13 answering agents on five base models, all served locally through Ollama: six llama3.2 agents scoring 0.434 to 0.487 on MMLU, four qwen2.5 agents scoring 0.756 to 0.825, and single agents on mistral, phi3.5 and gemma2. They answered four closed-book multiple-choice benchmarks, MMLU, SciQ, TruthfulQA-MC1 and ARC-Challenge, plus two larger MMLU harvests served through vLLM in half precision. Baselines included majority vote, Dawid-Skene, TruthFinder, CRH, CATD, the dependence-aware ACCU and DEPEN, correlation fusion, ClusterVote, the recent CARE, ISP and OW-I, and a best agent chosen using labels. Each dataset was split repeatedly into validation and test halves and compared with a corrected resampled t-test under Holm correction. Sub-pools and stress tests varied the pool's composition. Question and split counts were lost from the copy we read, so we do not quote them.

6. How it performed

On MMLU, BP-TD with provenance weights and no labels reached 0.716 against 0.624 for majority vote and 0.500 for Dawid-Skene; tuned on validation labels it reached 0.751, not significantly above correlation fusion at 0.743. The larger MMLU harvests kept a smaller significant gain, 0.718 against 0.681 and 0.707 against 0.674. No other benchmark showed a significant win: ARC 0.879 against 0.847, SciQ nearly flat, TruthfulQA 0.586 against 0.650. Reliability rankings were recovered on MMLU, where classical estimators inverted them, and on ARC. The gain reverses when the largest bloc is the strongest, and no label-free statistic tested tells the cases apart. With about twenty labels, picking the single best agent beat every aggregator on MMLU, ARC and TruthfulQA. The author notes that only local open models were tested and that benchmark contamination is possible. Our call: pilot it for auditing agent pools where labels are truly unavailable, and otherwise spend twenty labels on choosing the best agent. (Source: arXiv 2611.04805, 2026)