MLPerf Storage

Purpose

Storing and processing of training data is a crucial part of the machine learning (ML) pipeline. The way we ingest, store, and serve data into ML frameworks can significantly impact the performance of training and inference, as well as resource costs. However, even though data management can pose a significant bottleneck, it has received far less attention and specialization for ML. 

The main goal of the MLPerf Storage working group is to create a benchmark that evaluates performance for the most important storage aspects in ML workloads, including data ingestion, training, and inference. Our end goal is to create a storage benchmark for the full ML pipeline which is compatible with diverse software frameworks and hardware accelerators. The benchmark will not require any specific hardware for performing computation. 

Creating this benchmark will establish best practices in measuring storage performance in ML, contribute to the design of next generation systems for ML, and help system engineers find the right sizing of storage relative to compute in ML clusters.

Deliverables

• Storage access traces for representative ML applications, from the applications’ perspective. Our initial targets are Vision, NLP, and Recommenders. (Short-term goal)
• Storage benchmark rules for:
  • Data ingestion phase (Medium-term goal)
  • Training phase (Short-term goal)
  • Inference phase (Long-term goal)
  • Full ML pipeline (Long-term goal)
• Flexible generator of datasets:
  • Synthetic workload generator based on analysis of I/O in real ML traces, which is aware of compute think-time. (Short-term goal)
  • Trace replayer that scales the workload size. (Long-term goal)
    
• User-friendly testing harness that is easy to deploy with different storage systems. (Medium-term goal)Nunc auctor tempor ornare class sollicitudin