Benchmark Suite Results
test results view
The MLPerf Storage benchmark suite measures how fast storage systems can supply training data when a model is being trained. Below is a short summary of the workloads and metrics from the latest round of benchmark results submissions.
Results
MLCommons results are shown in an interactive table to enable you to explore the results. Choose a workload below to see its results. Every submission is shown in full: use the filters in the table to narrow by organization, division or availability, and select Show all metric columns to see every measured value rather than the summary set. Results can be downloaded from the toolbar at the bottom of the table.
Workloads
Each workload supported by MLPerf Storage is defined by a corresponding MLPerf Training benchmark. The following table summarizes the workloads in this version of the benchmark (the rules remain the official source of truth):
| Area | Task | Model | Nominal Dataset | Latest Version Available |
|---|---|---|---|---|
| Vision | Medical image segmentation | 3D U-Net | KITS 2019 (602x512x512) | v2.0 |
| Vision | Image classification | ResNet50 | ImageNet | v2.0 |
| Scientific | Cosmology parameter prediction | CosmoFlow | CosmoFlow N-body simulation | v2.0 |
| Language | Language processing | BERT-large | Wikipedia (2.5KB/sample) | v0.5 |
| Checkpointing | Preserve forward progress in the face of infrastructure failures during Training | Llama-3 | From 105GB at an 8B scale to 18TB at a 1T scale. | v2.0 |
The dataset is referred to as a “nominal dataset” above because the MLPerf Storage benchmark simulates the above named real datasets using synthetically generated populations of files where the distribution of the size of the files matches the distribution in the real dataset. The size of the dataset used in each benchmark submission is automatically scaled to a size that prevents significant caching of the dataset in the systems actually running the benchmark code.
Divisions
MLPerf aims to encourage innovation in software as well as hardware by allowing submitters to reimplement the reference implementations. There are two Divisions that allow different levels of flexibility during reimplementation:
- The Closed division is intended to allow comparisons between storage systems in an “apples-to-apples” fashion and requires using a fixed set of benchmark tunables and options when running the benchmark.
- The Open division is intended to foster innovation, to show how performance could be increased if some changes were made. As a result, it allows using different data storage formats, access methods, tunables, and options.
- See the rules for specifics on what can be changed in each Division.
Availability
MLPerf divides benchmark results into categories based on the availability of the storage solution:
- Available systems contain only components that are available for purchase or for rent in the cloud.
- Preview systems must be submittable as Available in the next submission round. Ie: the code or h/w you used is on a path to being Available and will likely be Available within the next 6 to 9 months, it just hasn't quite gotten there yet.
- Research, Development, or Internal (RDI) contain experimental, in development, or internal-use hardware or software. These may never be Available, either because they're a proof-of-concept, or you believe you'll need to change them before being released because you learned things, etc.
Submission Information
Not all of the below columns are part of every submission round.
Each row in the results table is a set of results produced by a single submitter using the same software stack and hardware platform. Each Closed division row contains the following information:
Each row in the results table contains the following information for each workload submitted:
Throughput
This is the maximum performance the storage system was able to deliver while maintaining all the accelerator(s) at 90% utilization or above (ie: no more than 10% of the time were the accelerator(s) idle and waiting for the storage system to deliver data). It is reported as both “samples/second”, a metric that should be intuitively valuable to AI/ML practitioners, and as “MB/s”, a metric that should be intuitively valuable to storage practitioners.
Number of Simulated Accelerators
The number of simulated accelerators active during this test; ie: how many accelerators of the given type can this storage system keep busy.
Dataset Size
Since the dataset used in this test was synthesized and must be of a size to prevent significant caching of data in the compute node(s) running the benchmark, the size of the dataset used in this test is reported here.