# AgentSWE benchmark tests what coding agents actually deliver

> The arXiv preprint defines 25 tasks across agent creation, editing existing code and optimization toward a target.

- Canonical URL: https://newsverum.com/en/news/agentswe-coding-agent-benchmark-requirement-fidelity-2026
- Publisher: Newsverum
- Section: Technology
- Country: China
- Published: 2026-10-13T10:04:09+00:00
- Updated: 2026-10-13T10:04:09+00:00
- Event / data period: 2026-10-09
- Other editions: [Türkçe](https://newsverum.com/tr/news/agentswe-coding-agent-benchmark-requirement-fidelity-2026.md), [العربية](https://newsverum.com/ar/news/agentswe-coding-agent-benchmark-requirement-fidelity-2026.md), [Français](https://newsverum.com/fr/news/agentswe-coding-agent-benchmark-requirement-fidelity-2026.md), [Español](https://newsverum.com/es/news/agentswe-coding-agent-benchmark-requirement-fidelity-2026.md), [Deutsch](https://newsverum.com/de/news/agentswe-coding-agent-benchmark-requirement-fidelity-2026.md), [Português](https://newsverum.com/pt/news/agentswe-coding-agent-benchmark-requirement-fidelity-2026.md)

Researchers affiliated with the Beijing Academy of Artificial Intelligence and partner institutions released AgentSWE, a benchmark for measuring whether a coding agent builds another agent that meets a natural-language requirement. The arXiv preprint dated October 9 covers 25 tasks in three stages: creating an agent from scratch, editing production agent codebases and optimizing toward a measurable target. Builders work with a limited set of illustrative cases; the delivered version is then evaluated on held-out cases using criteria derived from the stated requirements.

The authors report that none of the evaluated model and harness combinations came close to delivering what the requirements asked for. Their analysis identifies recurring problems including overfitting to visible examples, runtimes that do not perform the promised work and unsupported claims of completion. The benchmark therefore tests whether a built agent performs its stated job, rather than whether its code appears complete. These findings are specific to the 25 research tasks and evaluation setups in the preprint; they do not establish a universal failure rate for coding agents or production deployments.

## Data

### AgentSWE task and stage counts in the preprint

Task and lifecycle-stage counts defining benchmark scope.

Methodology: The paper assigns 25 tasks across creation, editing and optimization; counts apply only to this research benchmark.

| Measure | Value |
|---|---|
| AgentSWE tasks | 25 |
| Creation, editing and optimization stages | 3 |

Sources: <https://arxiv.org/abs/2610.13361> · Download data: [JSON](https://newsverum.com/api/v1/articles/agentswe-coding-agent-benchmark-requirement-fidelity-2026/data/benchmark-scope-counts?lang=en&format=json) · [CSV](https://newsverum.com/api/v1/articles/agentswe-coding-agent-benchmark-requirement-fidelity-2026/data/benchmark-scope-counts?lang=en&format=csv)

## Sources

- [arXiv preprint by researchers affiliated with Beijing Academy of Artificial Intelligence, Institute of Automation of the Chinese Academy of Sciences and Renmin University of China](<https://arxiv.org/abs/2610.13361>) (Source checked: 2026-10-13; Published: 2026-10-09)

---

Cite the canonical URL above. [Editorial standards and corrections](https://newsverum.com/en/editorial)