AgentSWE benchmark tests what coding agents actually deliver

Researchers affiliated with the Beijing Academy of Artificial Intelligence and partner institutions released AgentSWE, a benchmark for measuring whether a coding agent builds another agent that meets a natural-language requirement. The arXiv preprint dated October 9 covers 25 tasks in three stages: creating an agent from scratch, editing production agent codebases and optimizing toward a measurable target. Builders work with a limited set of illustrative cases; the delivered version is then evaluated on held-out cases using criteria derived from the stated requirements.

The authors report that none of the evaluated model and harness combinations came close to delivering what the requirements asked for. Their analysis identifies recurring problems including overfitting to visible examples, runtimes that do not perform the promised work and unsupported claims of completion. The benchmark therefore tests whether a built agent performs its stated job, rather than whether its code appears complete. These findings are specific to the 25 research tasks and evaluation setups in the preprint; they do not establish a universal failure rate for coding agents or production deployments.