MLCommons to develop an NSF-backed agent efficiency benchmark
MLCommons was selected to develop the MLPerf Agents benchmark under a two-year, $3.5 million AI Efficiency Challenge supported by the U.S. National Science Foundation. The planned benchmark will measure complete workflows—not just model token serving—including the agent harness, model choice, retrieval, tool execution, orchestration, state management, failure recovery and validation.
Event / data period:
Published: · Updated:
It will report cost, energy, latency and throughput for tasks that are successfully verified. Initial workloads are planned for software engineering, research and data analysis, and customer service. MLCommons says it will use common rules and hidden validation sets; the benchmark has not yet produced results. The announcement concerns a two-year development project.
Data
MLCommons to develop an NSF-backed agent efficiency benchmark
MLCommons was selected to develop the MLPerf Agents benchmark under a two-year, $3.5 million AI Efficiency Challenge supported by the U.S. National Science Foundation. The planned benchmark will measure complete workflows—not just model token serving—including the agent harness, model choice, retrieval, tool execution, orchestration, state management, failure recovery and validation. It will report cost, energy, latency and throughput for tasks that are successfully verified. Initial workloads are planned for software engineering, research and data analysis, and customer service. MLCommons says it will use common rules and hidden validation sets; the benchmark has not yet produced results. The announcement concerns a two-year development project.
- · Occurred
MLCommons was selected to develop the MLPerf Agents benchmark under a two-year, $3.5 million AI Efficiency Challenge supported by the U.S. National Science Foundation. The planned benchmark will measure complete workflows—not just model token serving—including the agent harness, model choice, retrieval, tool execution, orchestration, state management, failure recovery and validation.
MLCommons was selected to develop the MLPerf Agents benchmark under a two-year, $3.5 million AI Efficiency Challenge supported by the U.S. National Science Foundation. The planned benchmark will measure complete workflows—not just model token serving—including the agent harness, model choice, retrieval, tool execution, orchestration, state management, failure recovery and validation. It will report cost, energy, latency and throughput for tasks that are successfully verified. Initial workloads are planned for software engineering, research and data analysis, and customer service. MLCommons says it will use common rules and hidden validation sets; the benchmark has not yet produced results. The announcement concerns a two-year development project.
Methodology
This record summarizes the named primary source and its reported measurements; no independent test or survey verification was performed.
Updated:
Subjects and places
Subject proposals
- Artificial intelligenceSuggested
- Subject suggested from the title or summary: AImlcommons.org ↗