MLCommons has been selected as the Benchmarking Team for the STRIDE Ventures AI Efficiency Challenge, a program supported by the U.S. National Science Foundation Directorate for Technology, Innovation and Partnerships (NSF TIP). Through a $3.5 million award over two years, MLCommons will develop MLPerf Agents, a new benchmark that measures how efficiently agentic AI systems get real work done — from a delegated goal all the way to a verified outcome

As a nonprofit consortium composed of companies, academics, and nonprofits, we’re grateful for the opportunity to help accelerate commercial adoption of AI solutions that dramatically improve efficiency.

So here’s what we’re going to do. 

Why agents need a new benchmark

Benchmarks that measure the time, cost, and energy of inference remain important, but need to evolve to better account for how agents work.

Agentic AI is fundamentally different from a single model answering a single prompt. An agent works across many turns, calls tools, retrieves information, carries growing context, recovers from failures, and checks its own work. 

MLPerf’s existing benchmarks already capture some of this complexity. The MLPerf Inference working group is developing agentic inference benchmarks that reflect the demands agents place on model serving: multiple turns, tool calls, growing context, and cache reuse.

MLPerf Agents goes bigger and broader. It expands measurement to the complete workflow — the agent harness, model selection, agentic inference, retrieval, tool execution, orchestration, state management, failure recovery, and validation. That wider scope makes it possible to study how both harnesses and models affect end-to-end efficiency, not just how fast a model serves tokens.

What’s new in MLPerf Agents

MLPerf Agents will bring four new benefits:

  • Efficiency tied to success. Cost, energy, latency, and throughput are measured for successfully completed tasks, where success is verified using techniques influenced by MLCommons AI Risk and Responsibility working groups.
  • Whole-stack scope. Resources are counted across the entire agent stack, showing where inefficiencies arise.
  • Comparable results across architectures. Common workloads, tracing, measurement rules, and validation let buyers compare complete solutions.
  • Trustworthy verification. Hidden validation datasets and MLPerf’s review and audit process reduce benchmark gaming.

Initial workloads are planned to cover software engineering, research and data analysis, and customer service. The goal is to give buyers evidence for choosing efficient agentic systems, and to help developers find the improvements that cut cost and energy as agentic AI scales.

Built on eight years of MLPerf

MLPerf Agents will build on eight years of MLPerf benchmarks that the industry already relies on to compare AI systems, including MLPerf Inference and MLPerf Power. It will follow the same open, collaborative process: workloads, rules, and reference implementations developed by the MLCommons community, with results subject to peer review.

David Kanter, MLCommons founder and head of MLPerf, and Professor Vijay Janapa Reddi, co-founder and VP of MLCommons, will lead the work with the MLCommons community, with support from Qualcomm, NVIDIA, Intel, AT&T, Workato, Gimlet Labs, and others.

As agentic AI scales, we are just beginning to understand how to optimize across the harness, model calls, tool invocation, retries and more to deliver the best cost and energy efficiency. MLPerf Agents will give developers the evidence they need to find and fix those inefficiencies, and give buyers the evidence they need to choose efficient systems.” — Vijay Janapa Reddi, Board Member and Vice President of Research, MLCommons

About the AI Efficiency Challenge

The AI Efficiency Challenge is run by STRIDE Ventures, a national platform for breakthrough innovation launched by the U.S. National Science Foundation and Start2 Group. The Challenge is administered by Start2 Group under an Other Transaction Agreement with NSF. Learn more at stride-ventures.com/ai-efficiency-challenge.

This material is based upon work supported by the U.S. National Science Foundation.  Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the U.S. National Science Foundation.