K² · Artificial intelligence
The benchmarks judging AI coding agents don't even agree with themselves
The paper audits three repository-level performance-optimization benchmarks (GSO, SWE-Perf, and SWE-fficiency) that evaluate coding agents by applying patches to real repositories and comparing runtime against unoptimized baselines and reference patches.
ZC
ZS
Zhi Chen, Zhensu Sun, Yuling Shi et al.
5 authors · cs.SE, cs.AI
The 30-second scan
Run the exact same "correct" answer twice, on two different computers, and it should pass both times.
- The authors argue that leaderboard scores can conflate runtime instability, benchmark-specific scoring rules, and how many tasks are already solved by at least one public submission.
- The study replayed official reference patches for 740 code optimization tasks across four common types of Google Cloud machines.
- Most benchmark tasks can be replayed, but reference patches satisfy the original benchmark validity rules in every cross-machine replay for only 39/102 GSO tasks, 11/140 SWE-Perf tasks, and 411/498 SWE-fficiency tasks.
740 code optimization tasks replayed across four types of Google Cloud machinesReference patches passed every cross-machine replay for only 39/102 GSO tasksReference patches passed every cross-machine replay for only 11/140 SWE-Perf tasks