Tool-Calling LLM Agents Under Realistic MCP Faults: Scale Does Not Purchase Fault Tolerance: what it means for business leaders

A fault-injection proxy calibrated on real MCP incident reports cut tool-calling agents' task scores by 60 percent, and a six-times larger model held up no better than the small one, so robustness has to come from the agent framework.

1. What the method is

MCP-Injector is a proxy that sits between an agent and its MCP tool servers and injects faults into tool calls without changing the agent or the tools. It implements six fault operators, four on content (a few corrupted characters, wrong words substituted, a value whose type violates the schema, an explicit MCP error) and two on timing (a 30 second delay, a call that never returns), each under one of three persistence schedules: permanent, transient or intermittent. The mix of injected faults is calibrated to incidents mined from 57,240 issue and pull-request threads across 1,774 open-source MCP servers. The authors then ran a 1,350-cell experiment crossing two gpt-oss models, three agent frameworks and three fault conditions. The headline results are that calibrated faults cut the task-completion score by 60 percent, from 8.08 to 3.23 out of 10, and that the 120B model kept no larger share of its fault-free score than the 20B did.

2. Why the method was developed

Every published tool-use benchmark scores agents under the assumption that tools answer correctly and on time, so an agent can look excellent in evaluation and still fall apart when a server returns an error, hangs, or quietly sends back a wrong value. Infrastructure chaos engineering injects faults below this layer, at the network or the container, and never exercises the agent logic that reads outputs, retries and composes an answer. Earlier reliability benchmarks used hand-picked scenarios with no control over whether a fault recurs on every call or appears once and clears. The authors wanted a controlled way to ask two questions our own teams ask: which agent designs degrade gracefully, and whether paying for a larger backbone buys operational robustness. Mining real incident reports gives the fault mix an empirical basis instead of guesses, although the authors note that public issues over-represent persistent, reproducible failures.

3. Who should care

Anyone in our organisation building an agent on MCP tools should read this, starting with platform engineering and the teams that own our agent frameworks, because the choice of orchestration layer and its error handling turned out to matter more than the model. Site reliability and QA get a test harness that drops into staging behind one configuration flag with a reproducible seed, so an agent release can be stressed before it ships. Procurement and whoever evaluates vendor agents should note that a vendor's capability benchmark says nothing about what happens when a tool fails. Finance and product leads deciding whether to pay for a bigger model should know that in this study the larger one completed fewer tasks under faults, not more. Security should register the dual-use point: the same transparent proxy could be abused by anyone already positioned on our MCP path.

4. How the method works

The proxy intercepts the call_tool hook in the MCP client library, presenting itself as a server to the agent and as a client to the real server. Each call passes through four stages: observe the tool name and arguments, decide whether and which operator fires given the profile, the persistence state and a seeded random stream, act by forwarding the call and mutating the response or short-circuiting it, and record a structured log line tagged so cooperating loggers can tell synthetic faults from real ones. Timing faults and explicit errors are produced on the request side without contacting the server; content faults mutate the real response. The profile holds 18 entries, one per operator and persistence pair, weighted by the mined counts. A permanent entry fires on every later call once drawn, a transient entry fires once, and an intermittent entry is held for a few calls and fires with a small probability. The mined distribution is heavily permanent, 91.3 percent of the weight, and explicit errors alone carry 71.2 percent.

5. How it was evaluated

The study uses MCP-Bench with 25 single-server tasks drawn from 25 MCP servers, each presented as a fuzzy user prompt and scored by an LLM judge against a rubric covering fulfilment, grounding, tool choice and parameter accuracy. The backbones were gpt-oss-20B and gpt-oss-120B, served locally at their released MXFP4 precision with a 16,000 token context window and no per-model prompt tuning. Three stacks ran the same tasks: the MCP-Bench plan-and-execute loop, the OpenAI Agents SDK and a LangChain prebuilt agent. Each task ran with the proxy off, with the proxy passing calls through unchanged, and with the calibrated fault profile, over three seeds, giving 1,350 cells under a 3,000 second timeout and a bounded number of rounds. A gpt-4o-mini judge scored each completed run five times and the median was kept; runs that timed out or errored scored zero in the primary analysis. The passthrough condition was statistically indistinguishable from no proxy and added roughly 0.1 to 0.5 seconds per call.

6. How it performed

Under faults, completion fell from 95 to 100 percent to 88 to 97 percent, and the task-completion score dropped 60 percent, with planning effectiveness down 67 percent; among runs that completed under both conditions the drop was still 57 percent. Partial degradation was the typical outcome, in 56 percent of cells. Six percent of runs answered plausibly without using tools, mostly a baseline habit that faults expose rather than cause. LangChain completed the most tasks, the OpenAI SDK the fewest, and a mixed model put the extra loss of the MCP-Bench and OpenAI stacks at 0.93 and 0.84 points, which the authors call suggestive. The 120B completed 91.1 percent against 95.6 percent for the 20B, its resilience ratio was no better, and it drew more faults per cell, a candidate explanation rather than a proven one. The authors stress one model family, an easy benchmark, almost no transient faults fired and a noisy judge. Our call: pilot the injector against our own stacks in staging before any model upgrade. (Source: arXiv 2610.32555, 2026)