Marginal tool utility in agents

TL;DR

On 25 public APEX-SWE Observability tasks, we tested GPT-5.3-Codex and Gemini 3.1 Pro with three observability tools, the logs query tool alone, and no additional observability tool. Removing Mattermost and Plane cut observability-tool calls by 58% across the two models (308 to 130). Accuracy rose from 32% to 36% for GPT-5.3-Codex and stayed at 36% for Gemini 3.1 Pro.

An LLM judge labeled whether each tool call appeared useful to the agent’s patch. Logs often supplied evidence about the failure. Without the logs tool, accuracy fell to 24% and 20%, respectively. This comparison distinguishes useful evidence from a larger tool menu; the call labels themselves are estimates, not measured causal effects.

These are observability-tool calls, not all agent calls or tokens. The evaluation covers 25 public tasks and two models. It did not measure token use, elapsed time, or provider spend.