Investigation correctness with Claude Code

Claude Code identified the correct root cause in 19 of 22 incidents with Foam, compared with 14 of 22 without an additional tool and 12 of 22 with the Sentry MCP setup.

How we evaluated investigation correctness

Each incident came from a production failure and has a human-written answer key. Claude Code received the incident prompt and a repository worktree, then investigated and wrote a root cause analysis without user intervention. We ran it once per incident in each of three setups, using the same Claude Code model, Fable 5.1. ^[1]

A GPT-4o judge compared each answer with the human-written key. It counted an answer as correct only when it identified the same incident and precise technical cause and would lead an engineer to the same fix. The answer key and rubric were held fixed across the three setups. ^[1]

Results

With Foam, five incidents that Claude Code missed alone were scored correct, a 22.7 percentage point difference on this set. Foam's setup scored seven incidents higher than the Sentry setup. The result describes these tools and incidents; it is not a general product ranking. ^{[1], [2]}

Where the answers differed

The 22 incidents split into five groups. The incident numbers below follow the benchmark results. ^[2]

Incident 21 shows why checking the production state can matter. The error said an issue-solver run could not be found. Claude Code alone and Claude Code with Sentry attributed it to an incorrect database environment. With Foam, Claude Code followed the failing trace and identified that the referenced run ID had no corresponding document. The answer key described the missing document as the cause. ^[2]

In four incidents, the Sentry setup received an incorrect score where Claude Code alone received a correct one. In incident 19, for example, the error was a GitHub “repository not found” response. The Sentry setup attributed it to token permissions; the correct answer was a hardcoded default pointing to a repository that no longer existed. One run per setup cannot establish that access to Sentry caused the wrong answer, but the output shows how a plausible explanation can diverge from the underlying cause. ^[2]

Three incidents remained unsolved in every setup. They involved expected lock behavior mistaken for a bug, a rate limit reached after accumulated tool output, and a production container without the .git directory required by a preflight check. ^[1]

Conclusion

In this follow-up, Claude Code with Foam found the correct root cause in 19 of 22 incidents, five more than the same agent without an additional tool. On this set, access to production traces, logs, and metrics changed several diagnoses. The benefit was visible in cases where the initial error supported a plausible but incorrect explanation.

This is a small, single-run evaluation scored by a model against human-written answers. It does not establish the expected improvement across other incidents, models, or tool configurations. Repeated runs and an independent review of the scored answers would make that estimate stronger. The same protocol can be run again as more capable models arrive.

References