Introducing the MLPerf End-to-End RAG Inference Benchmark

The MLCommons MLPerf Inference Working Group is excited to introduce the first instance of a new End-to-End Retrieval-Augmented Generation (RAG) benchmark. By answering from documents retrieved at query time rather than from weights alone, RAG reduces hallucination and draws on current, private knowledge, which has made it one of the most common ways language models are deployed. A production RAG system is rarely a single model; it is a pipeline of several models that retrieve relevant documents and reason over them to produce a grounded answer.

A Retrieval-Augmented Generation (RAG) pipeline ingests and chunks source data, converts it into semantic vector embeddings for storage and retrieval, and then combines relevant retrieved data with a user query to generate an accurate, model-driven response. This benchmark measures this entire pipeline. It is built around two workloads:

An ingestion pipeline that builds the vector database from a document corpus.
A question-and-answering (QnA) pipeline answers queries over the vector database, iterating retrieve-and-reason steps across multiple hops until it has enough evidence.

Together they capture what governs a real RAG deployment — multiple models of different roles and sizes, served across an iterative loop — none of which a single-model benchmark can measure.

It is the first multi-component MLPerf inference benchmark to score an entire RAG pipeline end to end. This post covers what makes this workload unique, the dataset and task, the models and metrics, and the reference implementation.

Why End-to-End RAG?

RAG is increasingly popular and for many deployments, essential. By grounding answers in documents retrieved at query time, it reduces hallucination, keeps answers up to date, and lets a general model use proprietary or private knowledge it was never trained on — exactly the properties enterprises need.

By nature, RAG is a multi-component system, and its behavior emerges from how those components are composed and served together, not from any one of them alone. Single-model LLM benchmarks can’t capture this: they score one model on one prompt (most often without tokenzier/de-tokenizer involved), so the pipeline-specific behavior that governs a real RAG deployment goes unmeasured, and the optimization opportunities of serving several models together never arise.

It is also a stepping stone toward benchmarking agentic AI. RAG in itself is one of the tools an agent reaches for, and its multi-component, multi-model serving challenges are a first taste of what a full agentic benchmark must measure — laying groundwork that future agentic benchmarks can build on.

And it opens optimization avenues we haven’t seen in prior LLM benchmarks. Because the pipeline runs several models with different roles and sizes — rather than one model on one prompt — a submitter has levers that a single-model benchmark simply doesn’t expose. A few of them:

– Model placement. How the components are mapped onto the system — e.g. the smaller and larger language models on separate accelerators, or the lightweight components on CPU with varying precision levels without compromising E2E Accuracy criteria.

– Co-residency. How to share one device among multiple models — via memory partitioning or hardware-level isolation.

– Scheduling a multi-stage fleet of calls. How to overlap many concurrent tasks as an effective pipeline across heterogeneous accelerators involving macro and micro batching across the pipeline and within each ingredient in the pipeline.

– Prefix caching. The multi-hop context grows each hop, so reusing the shared prefix trades compute-bound prefills for memory-bound.

– System level optimization/KPI: Effective pipeline implementation across CPU/GPU/NIC/Storage to demonstrate system level KPI mimicking real world deployment scenarios

Dataset and task selection

The task is multi-hop queries answering over a set of Wikipedia documents, using the FRAMES benchmark:

824 queries, each with a ground-truth answer and the Wikipedia article URLs required to answer it.
2,515 Wikipedia HTML articles, a frozen snapshot shipped with the benchmark so every submitter indexes the same corpus.
~107,000 passages — the articles chunked into 768-character passages with 32 characters overlap for embedding and indexing.

FRAMES was chosen because it is public and factual, with unambiguous ground truth, and its queries span diverse reasoning types — numerical, tabular, temporal, multiple-constraint, and post-processing.

Above all, its queries are multi-hop, which is what makes FRAMES a demanding test of a RAG system. A multi-hop queries can’t be answered from one passage — it requires chaining several facts that never appear together, so every stage has to pull its weight. For example (Figure 1),

> “The person who posted a photo with Rahul Ligma and Daniel Johnson at the headquarters of a social media company claims to have a certain syndrome, despite never receiving a formal diagnosis. Who was this syndrome named after?” → ground truth: Hans Asperger

No single passage holds the answer. The pipeline has to chain three facts across separate articles: who posted that photo (Elon Musk), the syndrome he says he has (Asperger’s), and whom it is named after (Hans Asperger) — reformulating its searches hop by hop until the evidence connects.

Figure 1: Example Multi-hop E2E RAG

The benchmark exercises this dataset through two independent pipelines, each its own MLPerf workload:

Ingestion (`e2e-rag-db`) runs once to build the database; QnA (`e2e-rag-qna`) is the scored end-to-end pipeline.

Figure 2: Ingestion and QnA Pipeline for E2E RAG

The ingestion pipeline turns the corpus into a searchable vector database, which is a one-time operation. Parsing extracts the article body from each of the 2,515 Wikipedia HTML files shipped with the benchmark, stripping Wikipedia metadata such as references and navigation. Tables and lists are flattened into text row by row rather than dropped. Chunking slices that text into 768-character passages, each tagged with its original Wikipedia URL so it can be traced back to its page. The size was tuned: larger chunks mix in unrelated text, smaller ones split related facts apart. A 32-character overlap keeps facts that land on a boundary intact. Embedding encodes each passage into a 768-dimension vector, and the vectors are indexed into a FAISS HNSW graph for fast approximate similarity search. The result is the vector database that every QnA run queries against.

The QnA pipeline answers a query by looping over that database generated in the Ingestion pipeline (E2E-RAG-DB). Answering a query is not a single model call but a QnA task: the query runs several components through an iterative loop with sub-queries to reach an answer. A query rewriter first decomposes the query into up to three focused sub-queries. Each sub-query is embedded with the same embedder from ingestion, and used to retrieve its most similar passages from the database. The reranker then deduplicates these candidates and reorders them by relevance, keeping the strongest few. A document grader judges each retrieved passage for relevance and keeps only those that help, and a sufficiency checker decides whether the accumulated evidence can answer the query. If not, the query rewriter generates new sub-queries, reformulating and broadening its searches, and the loop repeats for up to five hops. Once the evidence is sufficient (or the hop limit is reached), answer generation produces the final answer grounded in the kept passages, or returns “Unknown” if the evidence falls short.

Model selection

Component	Model	Parameters	Ref. Precision	Layers	Vector Dim
Embedding	intfloat/e5-base-v2	110M	FP32	12	768 per passage
Reranking	ColBERTv2.0	110M	FP32	12	128 per token
Query Rewriter	GPT-OSS-120B	120B / 5.1B Active	MXFP4	36	–
Sufficiency Checker	GPT-OSS-120B
Final Answer Generation	GPT-OSS-120B
Document Grader	GPT-OSS-20B	20B / 3.6B Active	MXFP4	24	–
Judge	Llama-3.1-8B	8B	BF16	36	–

All models can be downloaded from MLCommons-Storage

intfloat/e5-base-v2 is the embedder. Trained for retrieval with separate query and passage encodings, it fits RAG’s query-to-document matching and performs strongly on standard benchmarks (MTEB, BEIR), all at a compact, widely adopted 110M parameters.
ColBERTv2.0 is the reranker, using late interaction rather than the typical cross-encoder. Its token-level matching captures finer-grained relevance than compressing a passage into one vector, and generalizes better to new domains, which suits the private document collections common in enterprise use.
GPT-OSS-120B handles the reasoning-heavy work of rewriting a query into good sub-queries, judging when the evidence is sufficient, and generating the final answer. It is open, and already widely deployed and well-optimized by MLPerf submitters.
GPT-OSS-20B covers document grading, deciding whether a retrieved passage is relevant. This is a high-volume classification task, and pairing a smaller model here reflects real deployments that route easy work to right-sized models rather than sending everything to the largest one. It also shares the gpt-oss architecture with the 120B, so supporting it takes little extra effort.
Llama-3.1-8B-Instruct is the judge, deliberately from a different family than the GPT-OSS models it grades to avoid self-preference bias. It scores answers after the accuracy run is finished.

Performance metrics

MLPerf Inference traditionally defines two serving scenarios, Offline and Server. This first instantiation focuses on Offline, where all requests are available at once and can be scheduled for maximum throughput; Server is left for future rounds. Each pipeline reports its own throughput metric: documents per second for ingestion, and tasks per second for QnA.

For ingestion, documents per second is the natural unit. Each document flows through parse, chunk, embed, and index, and the document is the fixed unit of work every submitter starts from, which makes documents per second directly comparable across systems. More details can be found in inference rules.

For QnA, the usual language-model metric, tokens per second, doesn’t fit: the pipeline runs two different-sized language models alongside non-language-model components, so a single token rate can’t represent the whole system. A per-hop metric like hops per second is hard to interpret, since a QnA task takes a variable number of hops. Tasks per second avoids both problems by measuring the unit that matters, one query answered end to end. It looks small next to other benchmarks because a single query can run up to five hops and a dozen or more language-model calls.

The QnA pipeline is also a complex, dynamic workload that is hard to make reproducible, on top of the inherent non-determinism of language-model generation. For performance runs, recorded reference inputs are provided for each stage of each hop, fixing the number of hops and the documents retrieved. Outputs are still generated normally but discarded, so the pipeline does real work while run-to-run variation stays minimal.

Accuracy metrics

Retrieval quality (against the required Wikipedia URLs) is not an official metric. It is a database-integrity check, used to confirm that a submitter’s independently built vector database behaves like the reference so that everyone is answering from the same corpus.

Reference values over the full 824-query set:

Metric	Value
Final answer	35%
Precision / Recall / F1	75% / 70% / 69%

A submission is valid if its answer accuracy is at least 97% of the reference accuracy.

Answer accuracy also varies with the kind of reasoning a query demands:

Answer accuracy also varies with the kind of reasoning a query demands. The breakdown below is insights only, and a query can carry more than one reasoning type.

Reasoning type	Answer accuracy
Multiple constraints	38%
Post-processing	34%
Temporal	32%
Tabular	31%
Numerical	31%

Multiple-constraint queries score highest, since they mainly require gathering discrete facts, exactly what dense retrieval and a language model are good at. Numerical and tabular queries are hardest: the answer depends on pulling exact figures out of prose or tables and then computing over them, so a single misread number or a chunk boundary that splits a table fails the whole query. Temporal and post-processing queries sit in between, needing date arithmetic or a final transformation on top of correct retrieval.

Closing these gaps points naturally toward a more agentic pipeline: a knowledge graph or structured index the system can query for entities and relations, table-aware parsing and chunking so figures survive ingestion intact, and equipping the answer step with tools such as a code interpreter to handle arithmetic and unit conversion. Each is a step from today’s fixed retrieve-and-read loop toward an agent that chooses how to find and process what it needs.

Compliance

E2E-RAG-QnA requires a single compliance test, TEST09 (output-token-length verification) which re-runs the workload in LoadGen performance mode with the audit.config and compares the mean output token length of the answer-generator response against the reference. The only component visible to LoadGen against the reference implementation’s mean OSL of 273.81 tokens with a ±10% band (246.43 – 301.19). This guards against a submission cutting work by generating systematically shorter answers in performance mode than the reference does.

The check is a single aggregate on one pipeline stage, it does not provide any guard to retrieval, reranking, or intermediate query/sufficiency LLM calls. Future revisions could add distribution-level or per-component compliance coverage.

Optimization Opportunities

E2E RAG pipeline is intentionally designed to expose multiple layers of optimization opportunities across the full inference stack from the Vector DB Generation following the rules algorithmic criteria to inference serving framework, with dense and sparse kernels to KV-cache-aware scheduling. The reference implementation provides a clear but unoptimized baseline in several of these dimensions, leaving room for submitters to explore system and kernel improvements while preserving accuracy.

CPU Pipeline: MLPerf Loadgen provides the entire 824 queries/tasks at one-shot (in this offline scenario) to the serving framework. This provides a great opportunity for the submitters to explore how to spawn the tasks across several cores of the CPU with appropriate affinity to their corresponding accelerators with memory placement strategy for the E2E pipeline and within each of the ingredients in the pipeline.
GPU Partitioning with KV Cache Balancing: Effective Multi-model placement with low latency within accelerators and across accelerators to saturate all the ingredient accelerators participating in compute/comms/storage. This includes efficient low-latency high throughput CPU-GPU interaction, GPU-GPU/Accelerator Interaction/Compute vs. COMMS Balancing across the macro & micro batching to keep a tight pipeline.
Multi-model with different precisions: All the selected models can be operated in different precision as long as the E2E Accuracy criteria is met. This provides greater innovation opportunity to selectively choose the right precision that can provide best performance at system level.

Conclusion

This is MLPerf’s first benchmark to measure an entire RAG pipeline end to end, rather than a single model in isolation. It captures what actually governs a deployed RAG system: multiple models of different roles and sizes, served together across an iterative, multi-hop loop.

This blazes the trail for the future Agentic RAG. The retrieve-reason-decide loop at its core is the same loop that drives agents, and the natural next steps — tool use, structured retrieval, more autonomy in how the system finds and processes information — move it steadily toward a full agentic benchmark. We look forward to evolving it with the community with EndPoints/Agentic Intercept in near future

References

Acknowledgements

We thank MLCommons E2E RAG Taskforce participants and the MLPerf Inference Working Group, MLCommons Platform Engineering team for their feedback, support and guidance in developing this benchmark. Our sincere appreciation to all the contributors across the industry & academia that was very instrumental to weigh-in evaluation of various proposals! We are grateful to everyone to make this first unique benchmark definition happen with collaboration across various time-zones.

We also thank the Multi-Turn Taskforce, whose ideas helped us address the non-determinism inherent in multi-hop benchmarking.

We are especially grateful to Satyapriya Krishna, one of the authors of the FRAMES dataset, who joined our discussions to offer clarifications and guidance.