Same Query, Different Answer: Run-to-Run Variance in Semantic Query Processing: what it means for business leaders

IBM researchers reran three semantic query benchmarks across 1.2 million model calls and found that at temperature 0 a few percent of row judgments flip, which query plans turn into different answers for most queries.

1. What the method is

This is a measurement study, not a new method. Two IBM researchers re-executed the text and tabular workloads of three semantic query benchmarks, SemBench, LRO-Bench and UDA-Bench, through the benchmarks' own harnesses at temperature 0, logged all 1.2 million model calls, and reanalysed SemBench's published executions. A semantic query engine lets SQL-style operators such as filter, map, join and aggregate send each row to a language model for judgment. The study measures how often the same query returns a different answer when run again with nothing changed. Its central result is a possible-worlds model: each row's judgment has some probability of flipping between runs, and the probability that a whole answer reproduces decays exponentially with the number of rows the answer exposes. From the measurements the authors derive reporting rules for benchmarks and design rules for engines.

2. Why the method was developed

Engines such as LOTUS, Palimpzest, Snowflake Cortex and BigQuery's model functions treat the language model as an oracle that gives one answer per row, and benchmarks compare them on one or a few executions at temperature 0, which makes decoding greedy and was assumed to make results repeatable. Earlier work had shown that model outputs vary at temperature 0 because inference servers batch requests and floating-point reductions reorder, but nobody had measured what that does inside a query plan, where hundreds of small judgments combine into one count, ranking or join. Hosted APIs give no determinism guarantee, deterministic modes on open-source servers cost throughput and still differ across GPUs, and prior benchmarks recorded no per-row outputs, so there was no way to tell whether a published comparison between two engines would survive a rerun. The authors set out to quantify that gap.

3. Who should care

Our data and analytics engineering teams are the first audience, since any pipeline that uses a model to classify, extract or join rows and then aggregates in SQL inherits this variance, and dashboards built on such pipelines will silently change between refreshes. Our platform and MLOps teams own the choice of serving stack, which the study shows changes the answer, its variance and its failure rate even with identical model weights. Procurement and anyone reading vendor benchmarks should know that a single-run comparison between two semantic query engines can reverse on the next run. Compliance and audit functions should care because a materialized column produced by a model is not reproducible unless the per-row outputs were recorded. Product managers shipping features such as sentiment counts or ranked lists built on model judgments should expect those numbers to move without any code change.

4. How the method works

The mechanism is statistical rather than algorithmic. Each row judged by an operator is treated as an independent draw from a distribution over outputs, so one execution is one possible world of a tuple-independent probabilistic database. For a plan that exposes every row, such as a per-row map, a full filter result or a complete join, the chance that two runs match is the product of each row's chance of agreeing, which falls exponentially with row count; a join over 1,560 pairs with a fraction of a percent of flips reproduces only one time in ten. Aggregates like COUNT change only with the spread of the count, since opposite flips cancel, and a LIMIT clause depends only on the rows up to the last qualifying one, so it absorbs changes. The authors also show which rows flip: those near a decision tie, which a single run's log-probability margin or reasoning-trace length predicts, because each call adds small noise, about a quarter of a nat, to the decision margin.

5. How it was evaluated

The study ran eight model deployments: gpt-oss-120B as a reasoning model, Llama-3.3-70B on both a managed service and a vLLM server, Qwen2.5-72B, Mistral-Small-24B, Granite-8B, and GPT-4o and Claude Sonnet 5 on a subset. SemBench's Movie and Medical text queries ran five to ten times each on LOTUS; LRO-Bench's five relational operators ran in 14 implementation settings, 735 query-setting combinations per deployment; UDA-Bench's Player set of 225 documents and 123 SQL queries ran five times. Prompts, plans, scoring and parsing were the benchmarks' own, with caching off, inputs checksummed, and every request hashed and logged with token counts and, where served, top-5 log-probabilities. Controls covered temperature 1, fixed seeds, concurrency, input order and reasoning effort. Joins used a seeded 40-review sample evaluated exactly rather than SemBench's sampled cascade. The paper text available to us is truncated before the repetition and calibration experiments, so those we know only from the abstract and summary.

6. How it performed

At temperature 0, between 0.07 and 10.2 percent of rows changed output over five runs depending on the model, 24 percent on a 24-way diagnosis task, and the commercial models sat inside that range at 7.5 percent for GPT-4o and 4.1 percent for Claude Sonnet 5. Plans amplified this as predicted: on UDA-Bench, 4 percent of extracted values changing flipped 67 percent of SQL answers, and only 48 percent of SemBench answers reproduced on average. In SemBench's published results, 21 percent of cells changed quality, deterministic plans included, and 16 of 28 tests of LRO-Bench's implementation recommendations were not settled by its runs. The same weights on two serving stacks gave different answers and failure rates of 59 against 9 percent. Per the abstract, voting reduced but did not remove variance, and recording outputs is the only full remedy. The authors flag limited executions and models as threats. Our call: act now, by recording per-row model outputs and requiring repeated runs before any engine comparison. (Source: arXiv 2610.13823, 2026)