MLPerf Agent
The MLPerf Agents working group develops the industry-standard benchmark suite for measuring the intelligence, speed, cost, and energy efficiency of deployed agentic AI systems. Across client devices, enterprise servers, and cloud data centers, the benchmarks evaluate complete single- and multi-agent runs from an initial task to a verified result across planning, model inference, memory and retrieval, tool execution, safety checks, and host resources.
Purpose
AI applications increasingly run as agents that plan, maintain state, retrieve information, call external tools, coordinate across sub-agents, and verify outputs across software, enterprise, scientific, and physical domains. In production, a fast model alone does not guarantee an efficient or reliable agent. Total time, cost, and energy depend on the complete run, where planning steps, growing context windows, memory and retrieval traffic, failed tool calls, retries, and queueing dominate the system budget.
A fair, reproducible standard must test agents in controlled environments and verify both that the task succeeded within policy limits and the full set of resources (e.g., hardware and software) spent to finish it. The MLPerf Agents working group brings together chip and cloud providers, model and agent builders, researchers, and enterprise users to guide full-stack engineering and enable clear comparisons of verified work completed per dollar, watt-hour, and second.
Deliverables
- Industry-standard benchmark rules, representative single- and multi-agent workloads, and outcome verification checks
- Reproducible task environments, reference agents, a benchmark runner, and system adapters that protect proprietary code and data
- A common execution trace and logging format that records planning steps, model calls, memory and retrieval, tool actions, retries, and system resource use
- Measurement and reporting standards for task quality, completion time and throughput, cost, and energy per verified task
- Submission rules, held-out test sets, result checkers, and audit procedures
- A public benchmark roadmap and peer-reviewed result publication
Meeting Schedule
Weekly for one hour. The day and time will be set at the working-group kickoff.
Working Group Chairs
David Kanter
David Kanter is a Co-Founder, Board Member, and Head of MLPerf at MLCommons, where he leads the MLPerf benchmark suites and working-group processes. He has more than sixteen years of experience in semiconductors, computer architecture, and machine learning systems, and has co-led MLPerf Inference, MLPerf Power, and MLPerf Endpoints. He holds a B.S. in Mathematics and Computer Science and a B.A. in Economics, both with honors, from the University of Chicago.
Vijay Janapa Reddi
Vijay Janapa Reddi is the Gordon McKay Professor of Electrical Engineering and Computer Science at Harvard University and a Co-Founder, Vice President, and Board Member of MLCommons. He has helped lead the MLPerf Inference, MLPerf Mobile, and MLPerf Tiny benchmarks across systems ranging from datacenter clusters to microwatt edge devices. His research focuses on computer architecture, physical AI, and efficient machine learning systems.