What is MLPerf Endpoints, and why it matters to buyers

What MLPerf Endpoints measures

MLPerf^® Endpoints measures what inference systems deliver in production-like conditions. Instead of benchmarking a tuned configuration in isolation, Endpoints measures a served endpoint: the model, hardware, serving software, and networking together, under realistic load, the way customers actually consume inference.

What buyers get

The full operating frontier, not a point. An Endpoint result is a Pareto frontier of operating points across concurrency levels, so you can see throughput and responsiveness at the load that matches your various workloads and SLAs, from low-concurrency interactive use to high-concurrency serving.

Metrics that map to decisions. These are the five primary metrics; anything else is a derived metric.

• Concurrency – simultaneous queries the system is serving, a rough proxy for the number of users
• Time to First Token P90 – how fast the vast majority (90%) of users see the first response; this measures the delay before a response starts 
• System Throughput in tokens per second – how much total work the system does
• System Throughput per Watt – Proxy for TCO of the system; uses provisioned power to normalize system throughput
• Interactivity in tokens per second per user – how fast each user’s answer streams

Results that keep pace with the market. Endpoints runs on rolling submissions with publication cohorts every few weeks, so results track current models and software instead of trailing them by months.

A trusted, peer-reviewed process. Results carry an explicit status: verified (through MLCommons peer review), provisional (submitted, passed initial checks, review pending), or unverified (never submitted for review). Buyers can weigh results accordingly and every result follows a defined audit process

What Endpoints shows you that other benchmarks do not

MLPerf Inference measures a system running a model in fixed scenarios, at a small set of predefined operating points for single-shot inference. MLPerf Endpoints measures the served endpoint the way it is actually bought and consumed: under production-like load, across the full operating range and for both single-shot and agentic / multi-turn tasks. That difference changes what a buyer can learn:

The whole curve, not fixed points. Endpoints publishes the full Pareto frontier across concurrency. Two systems with identical scores at a fixed point can behave very differently at your operating point, and only the curve shows it.

• Enterprise leaders: Reduce the risk of buying excess capacity or selecting infrastructure that cannot support business demand. Applications with comparable workload requirements can be evaluated at different points on the same tested system’s curve. 
• AI infrastructure decision-makers: Use measured load levels to assess capacity headroom and plan expansion around throughput and responsiveness requirements, rather than peak scores alone. 
• Model decision-makers: Assess whether the tested model and serving configuration can meet expected demand while maintaining the required user experience. This helps shortlist viable deployment options before committing integration effort and budget, with model quality remaining a separate selection criterion.

Measure the system as sold, not the configuration as tuned. Serving stack, batching, scheduling, and networking are included in the Endpoints measurement because they’re part of the product you are buying.

• Enterprise leaders: Reduce procurement and deployment risk by requesting results for the configuration being purchased, creating clearer expectations for supplier selection and acceptance testing. 
• AI infrastructure decision-makers: Evaluate hardware and serving software together, directing investment toward configurations that deliver useful capacity rather than assuming a faster accelerator guarantees a better service. 
• Model decision-makers: Assess how the selected model performs through the proposed serving environment, helping identify deployment choices that merit further validation before integration begins. Together, these decisions can reduce avoidable rework and strengthen production planning, while preserving the need for application-specific quality, security, and integration checks.

The tradeoff made visible. Every deployment trades throughput against per-user experience. Endpoints explicitly highlights that tradeoff; a fixed-scenario score cannot express it.

• Enterprise leaders: Make the cost–experience decision explicit by combining measured performance with actual pricing to assess the cost of delivering each application’s required responsiveness. 
• AI infrastructure decision-makers: Identify operating points that maximize throughput while meeting latency and streaming-speed requirements, improving utilization without planning around peak capacity that fails those requirements. 
• Model decision-makers: Use these performance limits alongside quality evaluations to determine which model and serving combinations are viable for each application. This supports a more defensible balance among inference spending, service quality, and model choice, with experience targets determined by business needs rather than a universal performance goal.

Results at market speed. Rolling submissions and publication cohorts every few weeks, instead of benchmark cycles measured in months, so results exist for the models you are evaluating today.

• Enterprise leaders: Reduce the risk of long-term purchasing decisions based on outdated performance assumptions, strengthening investment cases with newer evidence as relevant results become available. 
• AI infrastructure decision-makers: Use updated, comparable results to assess whether software improvements warrant retesting existing infrastructure before purchasing additional capacity. 
• Model decision-makers: Revisit deployment shortlists as supported models and serving stacks evolve, identifying options worth validating without assuming the newest release is automatically better. This supports faster adoption and better-informed renewal or upgrade decisions, while keeping benchmark compatibility, system availability, and application-specific testing central.

How to read Endpoints results

Four steps, buyer’s order of operations:

1. Locate your workloads. For example: interactive assistant, high-volume batch, agentic pipeline. Each maps to a concurrency region of the published curve.

2. Set SLAs (service level agreements) of your workloads based on the experience you need to provide to users. Pick the TTFT P90 and interactivity (TPS/user) your application needs. This eliminates operating points, not systems.

3. Compare throughput at your point. Among systems meeting your SLAs at your concurrency, compare per-system TPS. This is the capacity you are paying for.

4. Apply normalization. In MLPerf Endpoints, System Throughput is normalized by provisioned power, which is a good proxy for cost. You can also normalize by what you pay for: GPUs provisioned or $/GPU/hr. As a customer, you may be able to obtain the cost of a system or service directly from the vendor. As with all commercial arrangements, the price you pay depends on many factors, such as purchasing volume and relevant discounts, which are beyond our purview. Note that the only official normalization that MLCommons can guarantee is provisioned power; anything else is a derived metric (which is unverified).

How to use it in one sentence: Find the concurrency region that matches your workload, filter to systems that meet your latency and interactivity floor, then compare system throughput among what remains.

Why MLCommons Is Evolving towards MLPerf Endpoints

The shift to MLPerf Endpoints reflects how enterprise AI is actually bought, deployed, and consumed today. Traditional inference benchmarks evaluated isolated hardware performance using static, single-shot models under controlled, predefined conditions. However, modern AI buyers do not purchase raw silicon in a vacuum—they buy live, fully integrated serving stacks, APIs, and managed cloud endpoints designed to support dynamic, multi-turn workloads like AI agents. As enterprise workloads shift from simple queries to continuous production traffic, measuring peak hardware output no longer predicts true operational success.

By moving to MLPerf Endpoints, MLCommons provides executive decision-makers with a complete picture of real-world service delivery. Rather than evaluating isolated components, Endpoints measures the full production system as sold—capturing the combined impact of batching, scheduling, networking, and serving software across a full spectrum of real-time load conditions. This shift turns benchmarking from an abstract technical metric into a practical procurement tool, helping enterprise leaders accurately evaluate total cost of ownership, reduce over-provisioning risk, and ensure systems deliver smooth end-user experiences under peak demand.

The workloads: benchmarks, models, and datasets

Endpoints v1.0 ships with six benchmarks. Each one is the same recipe: a task (the kind of work), a model (the engine doing the work), and a dataset (the traffic the endpoint is measured under, plus the accuracy check that proves quality held at that speed). The six are two summarization (single-shot), one reasoning (single-shot), and three agentic coding (multi-turn).

Start from your workload, not from the leaderboard. Find the task family that matches what you are building, then read the curves for those benchmarks.

If your workload looks like…	Task family	Benchmarks to read
High-volume text processing: summarization, extraction, classification, document pipelines	Summarization (single-shot)	Llama 3.1 8B Instruct; GPT-OSS 120B
Hard problems with long “thinking” outputs: deep analysis, math and science, code generation	Reasoning (single-shot)	DeepSeek-R1
Agents that work in loops: coding agents, tool-calling assistants, multi-step automation	Agentic coding (multi-turn)	Qwen 3.6; Kimi K3; DeepSeek V4 Pro

Summarization (single-shot)

Llama 3.1 8B Instruct on CNN/DailyMail is the small dense model benchmark. 

• Read this curve if you run high-volume, cost-sensitive text work (summarization, extraction, classification) or plan to serve small fine-tuned models. 
• Carried forward from MLPerf Inference, so it also carries the longest comparison history. 
• Background: Small LLM inference in MLPerf 5.1

GPT-OSS 120B on the MLPerf GPT-OSS set is a mid-size mixture-of-experts (MoE) model, and the most popular benchmark in recent MLPerf Inference rounds. 

• The closest match if you are deploying a general-purpose open-weight model for chat, assistants, or document work; 
• Its popularity also means the broadest set of comparable submissions across vendors. 
• Background: GPT-OSS in MLPerf Inference

Reasoning (single-shot)

DeepSeek-R1 on the reasoning set is a large MoE reasoning model that produces long thinking outputs before it answers. 

• Read this curve if your workload is reasoning-heavy: complex analysis, math and science, code generation, or the planning steps inside agent systems. 
• Long outputs make sustained generation speed, not just time to first token, the number that matters. 
• Background: DeepSeek inference in MLPerf 5.1

Agentic coding (multi-turn)

Qwen 3.6, Kimi K3, and DeepSeek V4 Pro run on agentic tool-calling trajectories: three models measured on identical multi-turn agent traffic. 

• Agentic inference is the workload MLCommons expects to dominate going forward, and it could not be represented in the previous single-shot harness: each turn depends on the one before it, contexts grow as the agent works, and systems that reuse computation across turns pull ahead in ways single-shot tests never reveal. 
• Read these curves if you are buying capacity for coding agents or any tool-calling pipeline. Because all three models run the same traffic, you can also compare models against each other on the same agent workload, not just systems. 
• Background: Agentic inference for MLPerf

The four datasets:

In Endpoints, the dataset does two jobs. The performance set defines the traffic: prompt lengths, output lengths, and turn structure, so the load matches how the workload behaves in production. The accuracy set verifies the endpoint still produces correct answers at the measured speed, so throughput was not bought by degrading quality.

CNN/DailyMail (Summarization: Llama 3.1 8B Instruct). News articles paired with human-written summaries: the standard summarization corpus, with the traffic shape of document-processing pipelines (moderate inputs, short outputs). Downloads automatically with the benchmark.

MLPerf GPT-OSS set (Summarization: GPT-OSS 120B). A dedicated performance set (mlperf_gpt_oss_performance) drives the load; a companion accuracy set (mlperf_gpt_oss_accuracy) checks quality against AIME25 competition math, GPQA graduate-level science questions, and LiveCodeBench coding problems.

Reasoning set (Reasoning: DeepSeek-R1). Hard prompts drawn from AIME, MATH500, GPQA, MMLU-Pro, and LiveCodeBench. These problems force long chains of reasoning, so the benchmark stresses sustained token generation: the behavior that dominates reasoning and agent-planning workloads.

Agentic tool-calling trajectories (Agentic coding: all three models; the one agentic dataset). 613 multi-turn tool-calling trajectories, hosted in MLCommons storage, that replay how a real coding agent works: repeated model calls, tool results fed back in, context growing turn over turn. We score accuracy inline during the run, plus 200 tasks from SWE-bench Verified, where the agent must resolve real GitHub issues end to end.

The bottom line

MLPerf Endpoints measures the system you actually buy: the model, hardware, serving software, and networking together, under production-like load. You get the full Pareto frontier across concurrency, metrics that map to purchasing decisions, results that keep pace with the market, and a peer-review process you can trust.

Start with your workload, set your SLAs, and read the curve at your operating point. The result you get back is the capacity you are paying for.

Ready to use Endpoints in your next evaluation? Explore verified results and the submission guide at mlcommons.org/benchmarks/endpoints.