Every round of MLPerf® Inference is a snapshot of where the industry is investing its engineering energy, and v6.1 stands out on two fronts: it is the broadest field of submitters we have ever seen, and it marks a clear inflection toward agentic and end-to-end benchmarking alongside a wave of newly submitted hardware.
The highlights of this round include a record number of submitters, new accelerators that significantly improve per-device performance, and a new multi-turn benchmark. Read on for our results analysis.
Submitters and Submissions
This round brought a record 30 submitters spanning silicon vendors, system builders (OEMs and ODMs), cloud and neocloud providers, and specialized inference-software companies.
Several results this round were joint submissions, where two organizations collaborated: Dell_AMD, Dell_MangoBoost, RedHat_Intel, and RedHat_Supermicro.
The table below provides a breakdown of the systems and scores submitted by each organization:
Altogether, 120 systems across the Datacenter and Edge suites, in both the Closed and Open divisions, were submitted.
Benchmarks
MLPerf Inference benchmarks are divided into Datacenter and Edge categories. Each benchmark supports several scenarios, which may be one or more of: Offline, Server, Interactive, SingleStream, or MultiStream. Submissions can also be classified as either Closed or Open. Closed submissions require using a model that is mathematically equivalent to the reference implementation, essentially holding the model fixed to enable direct comparisons. For more details on the MLPerf Inference benchmark setup, visit https://mlcommons.org/benchmarks/inference-datacenter/.
MLPerf Inference v6.1 features 10 Datacenter and 6 Edge benchmarks, two of which are new: End-to-End RAG and Agentic Edge Inference. Two benchmarks have been updated: For VLM (Visual Language Model based on Qwen3), an Interactive scenario has been newly defined, and in GPT-OSS-120B, the interactive scenario has been updated to allow speculative decoding.
Datacenter — 10 benchmarks
Edge — 6 benchmarks
Bar length is scaled within each suite, so Datacenter and Edge bars are not comparable to one another.
MLPerf Inference submitters can choose to submit to any of the benchmarks. In the Datacenter category, the most popular benchmark is gpt-oss-120b, followed by Llama2-70b. This is the first time gpt-oss-120b has topped the list, after Llama2-70b was the most popular for the last few rounds. This demonstrates that the MLPerf benchmarking community (which highly values stability and comparability) is now fully embracing MoE models.
Performance improvement this round
To gauge performance improvements since the last round, we compared the best per-accelerator Offline and Server scores against those from v6.0.
Percentages are as reported in the results analysis. Bars are scaled per model, so lengths are comparable only within a row. rgat’s best Offline score sits marginally below v6.0.
The biggest gains are in VLM and DeepSeek R1 for both Offline and Server scenarios, significantly surpassing gains in other benchmarks. This is because those 2 benchmarks were submitted on a new Preview category system powered by Nvidia Vera Rubin. Other benchmarks’ top scores were achieved on the same hardware as in 6.0, so results reflect more gradual performance improvements from software stack and algorithmic optimizations.
Llama2-70b is the longest-running LLM in the MLPerf Inference suite, being introduced in early 2024, round v4.0. As such, it is a strong vehicle for showing LLM performance improvements over time. This is shown below in median per-accelerator performance for Server scenario submissions, which has improved 5.58x over 6 runs. Three main reasons contribute to the performance gains:
- Reduced numerical precision: Some v6.1 submissions use FP4 precision, whereas earlier rounds used FP8. Since MLPerf Inference has strong accuracy requirements, the performance gains did not come at the cost of accuracy loss.
- New generation of accelerators. Accelerators that were used in the initial rounds of Llama2-70b benchmarks have been replaced with newer, more performant ones in the latest rounds.
- Software performance improvements. Every submission round sees performance improvements even on the same hardware, a testament to the improvements our submitters bring and a reflection of continued improvement in the field.
A similar trend appears in the DeepSeek R1 benchmark, despite being around for less time. The figure below shows that best per-accelerator performance has increased by 2.7x in Offline and 5.7x in Server scenarios, respectively, within a year. Similarly, the Interactive scenario, which had only 2 submission rounds, increased by 2.7x over that period.
Results Highlights
Beyond performance improvements, results also contain additional important achievements. One continued trend is multi-node inference. In the last few rounds, the number of multi-node submissions has increased considerably, as shown in the figure below:
The upward trend started in round 4.1, 2 years ago, and this round reached an all-time record of 16 multi-node submissions. We also see the size of submitted systems increasing. Just last round, a record of 288 accelerators was set, only to be broken in this round by Cruose submissions 6.1-0026 and 6.1-0027, which used 512 accelerators. The latter of the submissions set a new record for the number of tokens/second generated in the MLPerf Inference benchmark by generating almost 5.8M tokens/second in GPT-OSS-120b offline test.
Another trend in MLPerf Inference is hybrid submissions that use different types of accelerators working together. These systems pose unique challenges because submitters must account for the different computational capabilities of the devices. This round contains two such submissions:
New Accelerators and System
v6.1 introduced a diverse set of newly submitted accelerators and systems — from on-device parts to a next-generation rack-scale platform — spanning multiple accelerator vendors and different scales of deployment.
Submitters in their own words
Many participating organizations submitted supplemental statements describing what they consider most significant about their v6.1 submission. Full statements are available in the results repository; here are the highlights.
“AMD is pleased to share its broadest inference submission to date, expanding from three models in v6.0 to seven models across three AMD Instinct GPU platforms: MI355X, MI350X, and MI350P.”
“Highlights include 113,455 tokens/s in the Llama 2-70B Server scenario… demonstrating consistently strong performance from high-concurrency interactive serving to peak batch processing.”
“For the past few years, serious agentic work meant a datacenter round trip. That assumption is what this submission is meant to retire.”
“Cisco submitted the benchmark’s first cross-vendor heterogeneous accelerator deployment, unifying eight NVIDIA H200 and eight AMD Instinct MI350X GPUs into one inference pool connected by a Cisco G200 Network.”
“With GB300 NVL72, CoreWeave achieved an aggregate total throughput of 1.16 million tokens per second, per-GPU throughput improved 17% in Offline… since MLPerf v6.0.”
“Over 5.5 million tokens per second on gpt-oss-120b and 2.9 million on DeepSeek-R1 on AMD MI355X, plus results on NVIDIA GB200, all submitted from Crusoe Managed Kubernetes and Managed Slurm.”
“The PowerEdge XE9785L with 8x AMD Instinct MI355X GPUs delivered exceptional performance across Llama2 70B and GPT-OSS-120B.”
“We successfully outperformed our results from the previous round… by approximately 50%.”
“This round of testing paired the B300 GPU with the 120-core Intel Xeon 6979P… Results became more interesting. Take a look for yourself.”
“Google intentionally focused the MLPerf 6.1 inference submission on DeepSeek-R1 as the industry transitions from standard LLMs to massive Mixture-of-Experts (MoE) architectures.”
“Two HPE Compute XD690 servers with NVIDIA Blackwell Ultra GPUs… achieved 136,000 tokens-per-second, and significantly, achieved an 8,500 tokens-per-second per GPU on DeepSeek-R1.”
“Intel Xeon 6 delivered its largest generational gains this round from software alone. On the same silicon and same socket count as v6.0, Llama3.1-8B Server throughput on Xeon 6980P rose 2.4× (+142%) and Offline throughput rose 56%.”
“These benchmark-proven platforms deliver top-tier AI performance, giving enterprise customers versatile liquid and air-cooling choices.”
“We submitted the first result from KRAI/Gentic, our automated agentic full-stack optimization framework… to automatically search for faster kernels and integrate them with vLLM.”
“Our benchmarks with Qwen3-VL-235B-A22B-Instruct and GPT-OSS-120B on the NVIDIA B200 and GB300 chips deliver extremely fast throughput, with GPT-OSS-120b particularly providing up to a 8.85% increase in throughput over the same workload during MLPerf Inference v6.0.”
“Mango LLMBoost delivered the first prefill/decode-disaggregated results on AMD Instinct GPUs. We also served the largest multi-region cluster ever submitted: four sites on two continents… running as one endpoint at 97% scaling efficiency.”
“Our G8825Z5, G4826Z5 and G4520G6 AI/HPC server series have demonstrated their capability, successfully running a comprehensive suite of demanding generative AI and Large Language Model architectures.”
“Nebius was one of two submitters with next-generation silicon.”
“The NVIDIA Vera Rubin NVL72 platform makes its MLPerf debut on two of the most challenging mixture-of-experts benchmarks, delivering up to 2.5x better throughput on DeepSeek-R1 and up to 3.7x better on Qwen3-VL than the prior generation.”
“In the Inference v6.1 benchmark, Oracle showcased performance across NVIDIA GB300 and B300, AMD Instinct MI355X, and Intel X12 platforms.”
“Our patent-pending inference technology applied to stock published model weights in NVFP4 precision with an FP8 KV cache… Llama-3.1-8B — 274,927 output tokens per second.”
“QCT submitted results based on two AI platforms addressing distinct deployment, performance, and scaling requirements.”
“This round we demonstrated efficient inference on two very different classes of hardware: Grace-Blackwell systems orchestrated by OpenShift and general-purpose Xeon CPUs, with the same open source vLLM serving both.”
“For the DeepSeek-R1 benchmark, the measured throughput reached 60,411 tokens/s in the Offline scenario and 59,668 tokens/s in the Server scenario, strictly under the accuracy and latency constraints defined by MLPerf.”
“In this Inference v6.1 benchmark, Supermicro provided results for the NVIDIA B300, AMD Instinct MI355X, Intel Arc Pro B70, and Intel Xeon 6/6+ CPUs.”
“TTA’s evaluation of the DeepGadget dg5W proves that enterprise-grade LLM serving no longer requires massive data center infrastructure, democratizing high-density AI for the Korean and global ecosystem.”
“Our eight-GPU B300 system delivered 53,267.2 samples/second on Whisper Offline… For Llama 3.1-8B, the same system achieved 156,867 tokens/second in Server and 161,767 tokens/second in Offline.”
“This node reached 63,991 tokens/second in the Offline scenario… p99 TTFT was 336 ms against a 3,000 ms limit, with accuracy above 83%.”
“Nearly 7,800 tokens/s on a single NVIDIA H200: this submission shows what Llama 3.1-8B can deliver when FP8 quantization and an optimized inference stack come together.”
Summary
MLPerf Inference v6.1 is one of the most expansive rounds in the benchmark’s history: 30 organizations submitted 120 systems across the Datacenter and Edge suites and both the Closed and Open divisions.
Just as notable as the volume is the range — from single-accelerator edge devices to rack-scale platforms with hundreds of accelerators, all measured under one consistent methodology. A few clear signals emerge.
The workload mix favors Mixture-of-Experts models
For the first time, GPT-OSS-120B is the most popular model, with DeepSeek-R1 and Qwen3-VL also drawing high participation.
Agentic and end-to-end benchmarking has arrived
A shift from single-shot benchmarking to agentic model evaluation.
New hardware appeared across the spectrum
From datacenter to edge devices, contributing to the best-ever per-accelerator scores seen this round.
Hybrid and scalable deployments are on the rise
The first-ever heterogeneous submission combining accelerators from different vendors, plus geographically distributed submissions spanning continents.
Inference is a scale-out workload
This round set a record for the largest-scale submission ever, at 512 GPUs — continued interest in scale-out inference deployments.
As always, these results are most valuable when read in context: the Closed division remains the foundation for apples-to-apples comparison, while the Open division surfaces the optimization techniques that often preview the mainstream. We encourage readers to consult both the results tables and the submitter statements.
A round like this reflects an enormous collective effort. Our thanks go to every submitting organization — established vendors and first-time participants alike — and to the many Inference Working Group members who build, review, and audit these benchmarks and keep MLPerf rigorous, relevant, and fair.