MLPerf Storage

What the Training benchmark measures

During model training, accelerators repeatedly read randomly ordered batches of samples from the training dataset. If storage cannot deliver samples as fast as the accelerators consume them, the accelerators sit idle — wasting the most expensive component in the system. The Training benchmark measures how many accelerators a storage system can keep busy.

The benchmark emulates the accelerators rather than requiring them: each simulated accelerator reads real data through a real ML framework (PyTorch) at the exact intensity of the real training job, then “computes” by sleeping for the measured per-batch computation time of the real accelerator (an NVIDIA B200 or an AMD MI355 in v3.0). The data path — storage through to client DRAM — is fully real; only the arithmetic is skipped. A result is valid only if the simulated accelerators stayed busy at least a threshold percentage of the time, called Accelerator Utilization (AU): 90% for Unet3D, 85% for RetinaNet. The reported figures are the average of five consecutive measured runs.

Two provisions prevent the benchmark from measuring the clients’ RAM instead of the storage: the dataset must be at least 5× the aggregate DRAM of all client nodes, and each run must process at least 500 batches per accelerator. For large accelerator counts, datasets reach many terabytes.

Column definitions

Read B/W and # Simulated Accelerators rise together: each simulated accelerator demands a roughly fixed bandwidth, so the bandwidth is approximately accelerators × per-accelerator demand × achieved AU.

Unet3D: large-file sequential reads

Unet3D is a 3D medical-image segmentation model, and its dataset (derived from the KiTS19 kidney-tumor CT scans) drives a distinctive I/O pattern:

A moderate number of very large files — averaging about 140 MiB each, with a wide spread (standard deviation ±65 MiB) — each holding exactly one training sample. Every file is read in full and sequentially; the next file to read is chosen effectively at random from the files not yet read this epoch. There are no writes.

Because the dataset must be 5× client DRAM, a submission at scale reads thousands to hundreds of thousands of these ~140 MiB files per epoch. Each simulated accelerator consumes samples in batches of 7 with 4 parallel reader threads, and the client software keeps a deep queue of file reads in flight, so the storage system sees many concurrent large sequential streams — a bandwidth-dominated workload. Keeping a single simulated B200 accelerator at 100% utilization requires about 5.9 GiB/s of read bandwidth, so the per-accelerator demand is severe: this is the most bandwidth-hungry workload in the suite.

What “good” looks like: high sustained aggregate read bandwidth under many concurrent sequential streams, without latency excursions long enough to stall any client’s pipeline below 90% utilization.

RetinaNet: small-file random reads at high rate

RetinaNet is an object-detection model trained on the OpenImages dataset, and it stresses the opposite end of the storage envelope:

A very large number of small files — about 315 KiB each (JPEG images), one sample per file — each read in full, in random order, with no writes. The 5×-DRAM sizing rule pushes the file count into the millions.

The per-accelerator bandwidth is modest — about 155 MiB/s for a simulated B200 — but each simulated accelerator opens and reads roughly 500 files per second (batches of 24, 8 reader threads). At hundreds of accelerators, the storage system must serve hundreds of thousands of small-file opens and reads per second from a namespace of millions of files. This makes RetinaNet a metadata- and IOPS-dominated workload: file-open latency, directory scalability, and small-request handling matter far more than streaming bandwidth.

What “good” looks like: sustained high file-operation rates with tightly bounded per-file latency, at 85%+ accelerator utilization.

Normalizing factors for training results

What the Checkpointing benchmark measures

Training a large language model takes weeks to months across thousands of accelerators, and hardware faults are routine at that scale. The defense is checkpointing: periodically writing the entire training state — model weights plus optimizer state — to storage, and reading it back after a failure. While a checkpoint is being written synchronously, training is stalled; while one is being restored, the whole cluster waits. Checkpoint bandwidth therefore converts directly into recovered training time.

The benchmark emulates checkpointing for four sizes of Llama-3-family model. Each size fixes the model architecture, the parallelism layout, and therefore the exact number of writing processes and bytes written — a CLOSED submission cannot change any of it:

The benchmark rules list these sizes nominally as 105 GB, 912 GB, 5.29 TB and 18 TB. A run performs 10 checkpoint writes, then 10 checkpoint reads of the same data. The read phase models recovery — including the case where a failed client’s checkpoint shard is read back by a different client, which is why the rules require the storage to be either simultaneously readable and writable across clients or rapidly remappable between them.

What the storage system experiences

Hundreds to a thousand processes, spread across the client nodes, simultaneously each stream one or two large files of their shard of the training state — roughly 9–20 GiB per process depending on the model — as fast as the storage will absorb them, with an fsync forcing every file to durable media before it counts as done. Five seconds later, they do it again; ten times in all. Then the same volume is read back in full, ten times.

It is a purely sequential, large-request workload — but a highly concurrent one, with hundreds of simultaneous streams, and it is one of only two workloads in the suite where write performance is measured. The fsync requirement means write-back caches cannot hide slow media: the clock stops only when data is durable. To keep client RAM from absorbing the working set, the rules require the data written per client node to exceed 3× that node’s memory (or caches to be explicitly dropped between the write and read phases).

For the two largest models the file structure changes slightly (model layers are written as many per-layer files by one data-parallel group while every process writes its optimizer shard), but from the storage system’s point of view the character is the same: massive parallel sequential writes, then massive parallel sequential reads.

Column definitions

Bandwidth and duration are two views of the same event — duration ≈ checkpoint size ÷ bandwidth — but duration is the operationally meaningful number (how long is training stalled?), while bandwidth is the one that normalizes across systems.

Normalizing factors for checkpointing results

What the Vector Database benchmark measures

Retrieval-augmented generation (RAG) and semantic search systems store documents as high-dimensional embedding vectors and answer queries by finding the stored vectors nearest to a query vector. At small scale the index fits in RAM; at production scale it does not, and the index moves to storage — making query performance a storage problem. This benchmark measures how well a storage system serves an on-disk approximate-nearest-neighbor (ANN) index under sustained query load.

The v3.0 CLOSED workload: a Milvus vector database holding 10 million vectors of 1,536 dimensions (float32 — about 6 KiB per vector, ~61 GB of raw vector data before index overhead), indexed with DiskANN, an index structure specifically designed to live on disk rather than in RAM. The measured phase issues top-10 similarity searches against that index across five consecutive timed runs. Search quality is not taken on faith: every reported result includes the recall of its answers, checked against an exact brute-force ground truth.

What the storage system experiences

After an initial bulk-load and index-build phase (not part of the measured result), the query phase is a stream of small random reads against a multi-hundred-gigabyte index structure. Each incoming query walks the DiskANN proximity graph: it hops node to node, and each hop that misses the client’s memory cache becomes a read of a small block of index data at an effectively random offset. Queries arrive continuously, so the storage sees a sustained, highly random, small-block, read-only pattern — latency-sensitive, since graph hops within one query are sequential in nature and storage latency accumulates per hop.

This is the suite’s purest random-read latency and IOPS workload, the opposite pole from checkpointing’s giant sequential streams.

Column definitions

Reading and normalizing these results

Read QPS and recall together, always. ANN search trades accuracy for speed: an index searched lazily returns fast, low-quality answers; searched thoroughly, it returns slow, high-quality ones. A row with higher QPS at meaningfully lower recall has not demonstrated faster storage — it has answered an easier question. Only rows with comparable recall are performance-comparable. The P99 latency belongs in the same joint reading: it is the tail cost of delivering that QPS at that recall.

What the KV Cache benchmark measures

When an LLM serves a conversation, it builds an internal KV cache — the attention keys and values for every token of context it has read. That cache is large and expensive to recompute, and GPU memory is far too small to hold caches for the thousands of concurrent conversations an inference service juggles. The emerging architecture offloads inactive KV caches to storage and reloads them when a conversation continues — turning LLM inference serving into a storage workload. This benchmark measures how well a storage system plays that role: its performance directly bounds how many concurrent conversations an inference cluster can serve and how quickly it responds when a user sends their next message.

The benchmark simulates a population of users conversing with an LLM — a mix of chatbot sessions, coding-assistant sessions and document-analysis sessions with realistic context lengths, response lengths and think times — and issues the resulting KV-cache traffic to real storage. Every CLOSED submission runs the same three fixed configurations back to back (three trials each, 300 seconds per trial, results averaged); each configuration becomes one group of four metric columns:

What the storage system experiences

Each simulated conversation’s KV cache is one file, written whole and re-read whole — never partially. The files are big: the KV state is 128 KiB per token of context for the 8B model and 320 KiB per token for the 70B model, so with contexts running from a few hundred tokens (chat) to 25,000 tokens (large coding sessions), individual objects range from tens of MiB to several GiB. A conversation turn writes its cache once — with an fsync before it counts — then reads it back repeatedly: once per batch of generated response tokens, and again in full each time the user returns for another turn. The byte mix is therefore strongly read-dominated, typically several times more bytes read than written, in whole-object sequential bursts arriving concurrently from every simulated user. The benchmark defeats the operating system’s page cache on reads, so a “Storage Only” read is a genuine storage read.

The workload thus combines checkpointing’s large sequential transfers with an inference service’s concurrency and latency sensitivity: many simultaneous multi-hundred-MiB reads whose completion time gates a simulated user’s response.

Each client node runs the full standard user population (200 users for the 8B Storage-Only test, 100 for Storage + Mem, 70 for the 70B test). Adding client nodes adds offered load — so the client-node count indicates how much aggregate demand the storage system absorbed while producing the numbers shown.

Column definitions

When comparing the three groups within one row: “Storage + Mem” throughput is expected to exceed “Storage Only” (the DRAM tier absorbs the hottest re-reads, and its Read B/W column counts only the traffic that reached storage), and the 70B group’s token throughput is expected to be well below the 8B groups’ (each token drags 2.5× the bytes, with fewer concurrent requests). These are three different tests, reported side by side.

Normalizing factors for KV cache results