French lab releases benchmark for long-horizon robot agents
MAGMA-Bench evaluates agents in simulated multi-robot tasks with changing goals and execution failures.
Event / data period: 2026-10-09
Published: · Updated:
Researchers from LAAS-CNRS and French company Siléane introduced MAGMA-Bench in a preprint dated 9 October. The benchmark evaluates persistent reasoning in long robotic tasks. It includes 12 scenario families with one to three robots, mission changes, interruptions, partial observations and simulated execution failures.
The evaluation contains 1,254 episodes. The authors say even the strongest models they tested fail on a substantial share of clean episodes, with success rates falling further under perturbations. A shared subset of 215 episodes was used to compare frontier models. Because the benchmark separates high-level decisions from perception and low-level control and relies on simulation, the results do not establish the performance of deployed robots.
Data
Study scope
Simulated episodes across 12 scenario families; not real-robot deployment
| MAGMA-Bench evaluation episodes | 1254 | 2026-10-09 |
Methodology
Count reported by the paper for its experimental scope; not a national statistic or field result.
Updated:
Subjects and places
Subject proposals
- Science and technologySuggested
- Preserved editorial category/topic; broad candidate onlyarxiv.org ↗
Geographic references
Only source-provided geographic references are shown. Opening an external map is optional. Optional story maps use OpenStreetMap. Opening a map sends your IP address and this site’s address to the map provider; location data remains readable as text without loading the map.
- France · Place role: Coverage · Place type: Country or area · Location precision: RegionalEvidence
- Explicit preserved editorial country/region; no city or national centroid inferredarxiv.org ↗