Today the MLPerf consortium released results for MLPerf Inference v0.7, the second round of submissions to their machine learning inference performance benchmark suite that measures how quickly a trained neural network can process new data for a wide range of applications on a variety of form factors.

MLPerf Inference v0.7 is an exciting milestone for the ML community. The second benchmark round more than doubles the number of applications in the suite and introduces a new dedicated set of MLPerf Mobile benchmarks along with a publically available smartphone application. The Inference v0.7 benchmark suite has been incredibly popular with 23 submitting organizations and over 1,200 peer-reviewed results - twice as many as the first round - for systems ranging from smartphones to data center servers. Additionally, this round introduces randomized third party audits for rules compliance. To see the results, go to mlperf.org/inference-results-0-7.

The MLPerf Inference v0.7 suite includes four new benchmarks for data center and edge systems:

MLPerf Mobile - A New Open and Community-driven Industry Standard

The second inference round also introduces MLPerf Mobile, the first open and transparent set of benchmarks for mobile machine learning. MLPerf Mobile targets client systems with well-defined and relatively homogeneous form factors and characteristics such as smartphones, tablets, and notebooks. The MLPerf Mobile working group, led by Arm, Google, Intel, MediaTek, Qualcomm Technologies, and Samsung Electronics, selected four new neural networks for benchmarking and developed a smartphone application. The four new benchmarks are available in the TensorFlow, TensorFlow Lite, and ONNX formats, and include:

“The MLPerf Mobile app is extremely flexible and can work on a wide variety of smartphone platforms, using different computational resources such as CPU, GPUs, DSPs, and dedicated accelerators,” stated Prof. Vijay Janapa Reddi from Harvard University and Chair of the MLPerf Mobile working group. The app comes with built-in support for TensorFlow Lite, providing CPU, GPU, and NNAPI (on Android) inference backends, and also supports alternative inference engines through vendor-specific SDKs. The MLPerf Mobile application will be available for download on multiple operating systems in the near future, so that consumers across the world can measure the performance of their own smartphones.

Additional information about the Inference v0.7 benchmarks will be available at mlperf.org/inference-overview.