Trial by TraceBenchmarking AI Agent Harnesses for Reliability and Failure Attribution

Abstract

Agent performance depends on both the underlying model and the harness that manages its context, tools, execution, and state. Most benchmarks evaluate task completion, leaving the reliability of the harness itself largely unmeasured.

We introduce TraceTrial-Bench: 95 expert-level, mechanism-oriented tasks spanning seven reliability dimensions. Trajectory Tracing captures native execution records, while Multi-source Evidence Acquisition combines the agent context, harness records, and environment state. A two-layer evaluation separately assesses Agent performance and Harness reliability; explicit mechanism chains support evidence-based failure attribution.

Across four harnesses, nine models, and 3,420 selected runs, macro-averaged Harness scores range from 59.8% to 72.0%. In 31.1% of runs, the task succeeds despite at least one failed Harness criterion. These results show why task outcomes and harness reliability need to be measured together.

@misc{tracelite2026trialbytrace,
  title  = {Trial by Trace: Benchmarking AI Agent Harnesses
            for Reliability and Failure Attribution},
  author = {Cao, Zhiyong and
            Zhu, Hengjie and
            Wang, Yongxin and
            Li, Xuchen and
            Li, Jintao and
            Liu, Shuxia and
            Liu, Dunqiang and
            Wang, Bo and
            Zhang, Kairan and
            Lin, Cong and
            Yu, JiangDi and
            Liu, Siyuan and
            Yao, Sijia and
            Wei, Yi-Liang and
            Ma, Lingxu and
            Wang, Aijia and
            Chen, Anxin and
            Wei, Zixian and
            Ruan, Xiang and
            Xu, Ruizhi and
            Shi, Ruqian and
            Xiao, Baihua and
            Zhou, Sicheng},
  year   = {2026},
  url    = {https://tracelite.ai/trial-by-trace.html}
}