01
The problem
Automated evaluation is assumed reliable for tool-using language agents because human annotation does not scale. That assumption had not been audited against human labels at scale.
04Research artifact · arXiv:2604.16706
agenthallu-bench — auditing agent evaluation
Code and data for “Auditing Automated Evaluation, Error Propagation, and Runtime Mitigation in Tool-Using Language Agents.” It audits the widely-assumed reliability of automated evaluation for tool-using LLM agents against human annotation.
Sole author · v1 April 2026 · revised 9 August 2026 · arXiv preprint · 11 pages, 4 figures, 8 tables
- 14,750
- execution traces
- 13
- LLM agents audited
- 4
- task domains
- 9 + 4
- proprietary + open-weight models
In short
- Introduces AgentProp-Bench: 14,750 execution traces from 13 LLM agents — 9 proprietary, 4 open-weight — across 4 domains.
- Shows substring-heuristic judging is inadequate for scoring tool-using agents, which undercuts a great deal of published agent evaluation.
- Covers error propagation through multi-step traces and runtime mitigation, not just end-state scoring.
- ACM classes I.2.7, I.2.6, H.3.4.
02
AgentProp-Bench
- 14,750 execution traces collected from 13 LLM agents — 9 proprietary, 4 open-weight.
- Four task domains, so findings are not an artifact of one benchmark's quirks.
- Traces, not just final answers, which is what makes error-propagation analysis possible.
03
Findings
Substring-heuristic judging — the default in much of the agent-evaluation literature — is inadequate. The paper audits automated evaluation against human annotation, traces how errors propagate through multi-step agent execution, and examines what runtime mitigation can actually recover.
04
Artifact
Code and data are public at github.com/bhaskargurram-ai/agenthallu-bench. 11 pages, 4 figures, 8 tables; v1 April 2026, revised 9 August 2026.