{"article":{"annotation":{"article_id":"agentswe-coding-agent-benchmark-requirement-fidelity-2026","available_at":"2026-10-13T10:04:14.046839Z","geography_revision":"ef6ee0010dda5fe7b240fd514749b43e214c57da73222d2e40ec83b276dd0977","id":"373edecc3665c8d850f02c49243aff5bf45e7c9cad498bf2fb8d409dc4ca0afb","input_sha256":"2acf660c78420155a44f9451943932bc3e1f9c081d93f72109720345b68caf7c","limitations":["Rule-derived subjects are candidates, not verified editorial labels or calibrated model probabilities.","Coverage is distinct from event location. Unknown country/city/coordinates remain absent; no automatic geocoding.","Article geometries keep their stated precision and methodology; reproducing them is not independent coordinate verification."],"method":"controlled-rules-v1","places":[{"admin1":null,"city":null,"coordinate_note":null,"coordinate_reference_system":"OGC:CRS84","coordinate_source":null,"country":"CN","evidence":[{"field":"country","language":"en","note":"Explicit preserved editorial country/region; no city or national centroid inferred","source_urls":["https://arxiv.org/abs/2610.13361"]}],"geometry":null,"id":"iso3166-1:CN","kind":"country-or-area","labels":{"ar":"الصين","en":"China","es":"China","fr":"Chine"},"match_status":"candidate","precision":"regional","role":"coverage","uncertainty_m":null}],"rules_sha256":"fb5ad3c27c91cb840f7a0a05f7f64a5cb26ad0374409ff9457f277c7092e22b4","schema_version":1,"subjects":[{"concept_id":"medtop:13000000","confidence":null,"confidence_method":null,"evidence":[{"field":"category+topics","language":"en","note":"Preserved editorial category/topic; broad candidate only","source_urls":["https://arxiv.org/abs/2610.13361"]}],"method":"controlled-rules-v1","rule_id":"technology-broad","status":"candidate"}],"taxonomy_revision":"1f2db31d811fca07e99b16d402c10e2cc2965c9df1c7b41d9be348b40a24e7fb","taxonomy_version":"IPTC Media Topics 2026-Q2"},"api_url":"https://newsverum.com/api/v1/articles/agentswe-coding-agent-benchmark-requirement-fidelity-2026?lang=en","category":"technology","content":{"key_points":[],"paragraphs":["Researchers affiliated with the Beijing Academy of Artificial Intelligence and partner institutions released AgentSWE, a benchmark for measuring whether a coding agent builds another agent that meets a natural-language requirement. The arXiv preprint dated October 9 covers 25 tasks in three stages: creating an agent from scratch, editing production agent codebases and optimizing toward a measurable target. Builders work with a limited set of illustrative cases; the delivered version is then evaluated on held-out cases using criteria derived from the stated requirements.","The authors report that none of the evaluated model and harness combinations came close to delivering what the requirements asked for. Their analysis identifies recurring problems including overfitting to visible examples, runtimes that do not perform the promised work and unsupported claims of completion. The benchmark therefore tests whether a built agent performs its stated job, rather than whether its code appears complete. These findings are specific to the 25 research tasks and evaluation setups in the preprint; they do not establish a universal failure rate for coding agents or production deployments."],"sections":[]},"corrections":[],"country":"CN","data":{"blocks":[{"columns":[{"key":"measure","label":{"ar":"القياس","de":"Messgröße","en":"Measure","es":"Medición","fr":"Mesure","pt":"Medição","tr":"Ölçüm"},"language":"en","type":"text"},{"key":"value","label":{"ar":"القيمة","de":"Wert","en":"Value","es":"Valor","fr":"Valeur","pt":"Valor","tr":"Değer"},"type":"number","unit":"count"}],"description":{"ar":"عدد المهام ومراحل دورة التطوير المحددة لنطاق المعيار.","de":"Anzahl der Aufgaben und Lebenszyklusphasen, die den Benchmark-Umfang festlegen.","en":"Task and lifecycle-stage counts defining benchmark scope.","es":"Número de tareas y etapas del ciclo de vida que delimitan el benchmark.","fr":"Nombre de tâches et d’étapes du cycle de vie définissant le périmètre du benchmark.","pt":"Número de tarefas e etapas do ciclo de vida que definem o escopo do benchmark.","tr":"Kıyaslamanın kapsamını belirleyen görev ve yaşam döngüsü aşaması adetleri."},"id":"benchmark-scope-counts","methodology":{"ar":"توزع الورقة 25 مهمة على الإنشاء والتحرير والتحسين؛ وتنطبق الأعداد على هذا المعيار البحثي فقط.","de":"Die Studie verteilt 25 Aufgaben auf Erstellung, Bearbeitung und Optimierung; die Zahlen gelten nur für dieses Forschungsbenchmark.","en":"The paper assigns 25 tasks across creation, editing and optimization; counts apply only to this research benchmark.","es":"El artículo distribuye 25 tareas entre creación, edición y optimización; las cifras corresponden solo a este benchmark de investigación.","fr":"L’article répartit 25 tâches entre création, modification et optimisation; ces nombres concernent uniquement ce benchmark de recherche.","pt":"O artigo distribui 25 tarefas entre criação, edição e otimização; os números se aplicam apenas a este benchmark de pesquisa.","tr":"Makale 25 görevi oluşturma, düzenleme ve optimizasyon aşamalarına ayırır; sayımlar yalnızca bu araştırma kıyaslamasına aittir."},"rowTranslations":[{"measure":{"ar":"مهام AgentSWE","de":"AgentSWE-Aufgaben","en":"AgentSWE tasks","es":"Tareas de AgentSWE","fr":"Tâches AgentSWE","pt":"Tarefas do AgentSWE","tr":"AgentSWE görevleri"}},{"measure":{"ar":"مراحل الإنشاء والتحرير والتحسين","de":"Phasen Erstellung, Bearbeitung und Optimierung","en":"Creation, editing and optimization stages","es":"Etapas de creación, edición y optimización","fr":"Étapes de création, modification et optimisation","pt":"Etapas de criação, edição e otimização","tr":"Oluşturma, düzenleme ve optimizasyon aşamaları"}}],"rows":[{"measure":"AgentSWE tasks","value":25},{"measure":"Creation, editing and optimization stages","value":3}],"sourceUrls":["https://arxiv.org/abs/2610.13361"],"title":{"ar":"أعداد المهام والمراحل في مسودة AgentSWE","de":"Anzahl der Aufgaben und Phasen im AgentSWE-Preprint","en":"AgentSWE task and stage counts in the preprint","es":"Recuento de tareas y etapas de AgentSWE en el preprint","fr":"Nombres de tâches et d’étapes d’AgentSWE dans le préprint","pt":"Contagem de tarefas e etapas do AgentSWE no preprint","tr":"Ön baskıdaki AgentSWE görev ve aşama sayıları"},"type":"table","updatedAt":"2026-10-13T09:31:52Z"}],"version":1},"data_downloads":[{"formats":{"csv":"https://newsverum.com/api/v1/articles/agentswe-coding-agent-benchmark-requirement-fidelity-2026/data/benchmark-scope-counts?lang=en&format=csv","json":"https://newsverum.com/api/v1/articles/agentswe-coding-agent-benchmark-requirement-fidelity-2026/data/benchmark-scope-counts?lang=en&format=json"},"id":"benchmark-scope-counts"}],"data_types":["table"],"editorial_illustration":{"alt":"Conceptual illustration of technology, processors and connected information.","asset_id":"newsverum-v1-technology","avif_variants":[{"height":320,"url":"https://newsverum.com/assets/newsverum/illustrations/v1/technology-480.avif","width":480},{"height":480,"url":"https://newsverum.com/assets/newsverum/illustrations/v1/technology-720.avif","width":720},{"height":640,"url":"https://newsverum.com/assets/newsverum/illustrations/v1/technology-960.avif","width":960},{"height":800,"url":"https://newsverum.com/assets/newsverum/illustrations/v1/technology-1200.avif","width":1200},{"height":1024,"url":"https://newsverum.com/assets/newsverum/illustrations/v1/technology-1536.avif","width":1536}],"credit":"Newsverum","documentary":false,"height":1024,"kind":"symbolic-editorial-illustration","language":"en","scope":"topic","url":"https://newsverum.com/assets/newsverum/illustrations/v1/technology-1536.webp","variants":[{"height":320,"url":"https://newsverum.com/assets/newsverum/illustrations/v1/technology-480.webp","width":480},{"height":640,"url":"https://newsverum.com/assets/newsverum/illustrations/v1/technology-960.webp","width":960},{"height":1024,"url":"https://newsverum.com/assets/newsverum/illustrations/v1/technology-1536.webp","width":1536}],"width":1536},"event_id":"event-f7ee1135-d06c-53b1-b4b9-7151383bd32a","event_period":"2026-10-09","event_period_basis":"event-or-reference-period","id":"agentswe-coding-agent-benchmark-requirement-fidelity-2026","image":null,"importance":0.6,"language":"en","modified_at":"2026-10-13T10:04:09+00:00","publication_basis":"publication-record","published_at":"2026-10-13T10:04:09+00:00","reading_minutes":1,"region":"world","section_illustration":{"alt":"Conceptual illustration of technology, processors and connected information.","asset_id":"newsverum-v1-technology","avif_variants":[{"height":320,"url":"https://newsverum.com/assets/newsverum/illustrations/v1/technology-480.avif","width":480},{"height":480,"url":"https://newsverum.com/assets/newsverum/illustrations/v1/technology-720.avif","width":720},{"height":640,"url":"https://newsverum.com/assets/newsverum/illustrations/v1/technology-960.avif","width":960},{"height":800,"url":"https://newsverum.com/assets/newsverum/illustrations/v1/technology-1200.avif","width":1200},{"height":1024,"url":"https://newsverum.com/assets/newsverum/illustrations/v1/technology-1536.avif","width":1536}],"credit":"Newsverum","documentary":false,"height":1024,"kind":"symbolic-editorial-illustration","language":"en","scope":"section","url":"https://newsverum.com/assets/newsverum/illustrations/v1/technology-1536.webp","variants":[{"height":320,"url":"https://newsverum.com/assets/newsverum/illustrations/v1/technology-480.webp","width":480},{"height":640,"url":"https://newsverum.com/assets/newsverum/illustrations/v1/technology-960.webp","width":960},{"height":1024,"url":"https://newsverum.com/assets/newsverum/illustrations/v1/technology-1536.webp","width":1536}],"width":1536},"slug":"agentswe-coding-agent-benchmark-requirement-fidelity-2026","sources":[{"checkedAt":"2026-10-13","name":"arXiv preprint by researchers affiliated with Beijing Academy of Artificial Intelligence, Institute of Automation of the Chinese Academy of Sciences and Renmin University of China","publishedAt":"2026-10-09","url":"https://arxiv.org/abs/2610.13361"}],"summary":"The arXiv preprint defines 25 tasks across agent creation, editing existing code and optimization toward a target.","text":"The arXiv preprint defines 25 tasks across agent creation, editing existing code and optimization toward a target.\n\nResearchers affiliated with the Beijing Academy of Artificial Intelligence and partner institutions released AgentSWE, a benchmark for measuring whether a coding agent builds another agent that meets a natural-language requirement. The arXiv preprint dated October 9 covers 25 tasks in three stages: creating an agent from scratch, editing production agent codebases and optimizing toward a measurable target. Builders work with a limited set of illustrative cases; the delivered version is then evaluated on held-out cases using criteria derived from the stated requirements.\n\nThe authors report that none of the evaluated model and harness combinations came close to delivering what the requirements asked for. Their analysis identifies recurring problems including overfitting to visible examples, runtimes that do not perform the promised work and unsupported claims of completion. The benchmark therefore tests whether a built agent performs its stated job, rather than whether its code appears complete. These findings are specific to the 25 research tasks and evaluation setups in the preprint; they do not establish a universal failure rate for coding agents or production deployments.","title":"AgentSWE benchmark tests what coding agents actually deliver","topics":[],"translations":[{"api_url":"https://newsverum.com/api/v1/articles/agentswe-coding-agent-benchmark-requirement-fidelity-2026?lang=tr","language":"tr","url":"https://newsverum.com/tr/news/agentswe-coding-agent-benchmark-requirement-fidelity-2026"},{"api_url":"https://newsverum.com/api/v1/articles/agentswe-coding-agent-benchmark-requirement-fidelity-2026?lang=en","language":"en","url":"https://newsverum.com/en/news/agentswe-coding-agent-benchmark-requirement-fidelity-2026"},{"api_url":"https://newsverum.com/api/v1/articles/agentswe-coding-agent-benchmark-requirement-fidelity-2026?lang=ar","language":"ar","url":"https://newsverum.com/ar/news/agentswe-coding-agent-benchmark-requirement-fidelity-2026"},{"api_url":"https://newsverum.com/api/v1/articles/agentswe-coding-agent-benchmark-requirement-fidelity-2026?lang=fr","language":"fr","url":"https://newsverum.com/fr/news/agentswe-coding-agent-benchmark-requirement-fidelity-2026"},{"api_url":"https://newsverum.com/api/v1/articles/agentswe-coding-agent-benchmark-requirement-fidelity-2026?lang=es","language":"es","url":"https://newsverum.com/es/news/agentswe-coding-agent-benchmark-requirement-fidelity-2026"},{"api_url":"https://newsverum.com/api/v1/articles/agentswe-coding-agent-benchmark-requirement-fidelity-2026?lang=de","language":"de","url":"https://newsverum.com/de/news/agentswe-coding-agent-benchmark-requirement-fidelity-2026"},{"api_url":"https://newsverum.com/api/v1/articles/agentswe-coding-agent-benchmark-requirement-fidelity-2026?lang=pt","language":"pt","url":"https://newsverum.com/pt/news/agentswe-coding-agent-benchmark-requirement-fidelity-2026"}],"url":"https://newsverum.com/en/news/agentswe-coding-agent-benchmark-requirement-fidelity-2026"},"version":"1"}