Today, MLCommons® announced the release of an updated version 4.0 of our MLPerf® Mobile benchmark application and the publication of results from the updated app.

MLPerf Mobile v4.0

The latest revision of the MLPerf Mobile app features several notable improvements.

First, MLPerf Mobile v4.0 includes a new test based on MobileNetV4, a state-of-the-art image classification model from Google Research. The MobileNetV4-Conv-L model boasts an impressive 83% accuracy with the ImageNet dataset, versus 76% accuracy for the prior standard, MobileNetEdgeTPU.

MobileNetV4-Conv-L is designed to perform well across a range of mobile processor types, from CPUs and GPUs to neural accelerators. The MLPerf Mobile working group worked closely with the MobileNetV4 team in order to ensure optimized performance. This combination of an improved model architecture and collaborative optimization has proven quite potent. Although MobileNetV4-Conv-L executes six times the number of mathematical operations of its predecessor, MobileNetEdgeTPU, benchmark execution times have only increased by a factor of roughly 4.6. More details about the model can be found in the MobileNetV4 paper.

MLPerf Mobile v4.0 incorporates support for accelerated machine learning (ML) inference across a range of popular mobile systems on a chip (SoCs) and the devices based on them. New in this release is support for independent hardware vendor (IHV)-provided hardware acceleration paths on the MediaTek Dimensity 9300 and 9300+ SoCs, Qualcomm Snapdragon 7/8/8s Gen 3 chips, and Samsung Exynos 2400 SoCs. These new additions join a number of already-supported SoCs. Furthermore, MLPerf Mobile v.4.0 now runs on even more Android-based devices via its TensorFlow Lite fallback path.

In addition to expanded device support, MLPerf Mobile v4.0 features an improved user experience courtesy of a refreshed and friendlier user interface and new configuration options that offer more control over exactly how the benchmarks run.

“Today, we celebrate four years and eight submissions since the inception of the MLPerf Mobile benchmark,” said Mostafa El-Khamy, co-chair of the MLPerf Mobile working group. “The MLPerf Mobile benchmark suite is continuously evolving. Each of the vision tasks–classification, object detection, and segmentation–has been updated since the first version of the benchmark application, and a new super-resolution task was added last year. Looking ahead, the MLPerf Mobile group is working to add generative AI tasks to a future version of the benchmark suite.”

The MLPerf Mobile app is available for download on GitHub for device and chip manufacturers and others interested in using the benchmark. Please see the release notes for a complete list of supported devices, download links, and a full accounting of the updates and new features in v4.0.

MLPerf Mobile v4.0 results

MLCommons has published a new set of benchmark results from Qualcomm Technologies, Inc. and Samsung based on the MLPerf Mobile v4.0 application. To view the results, visit the MLPerf Mobile results page.

We encourage additional participation to continue to help shape the MLPerf Mobile benchmark suite. To contribute, please join the MLPerf Mobile working group.

About MLCommons

MLCommons is the world leader in building benchmarks for AI. It is an open engineering consortium with a mission to make AI better for everyone through benchmarks and data. The foundation for MLCommons began with the MLPerf benchmarks in 2018, which rapidly scaled as a set of industry metrics to measure machine learning performance and promote transparency of machine learning techniques. In collaboration with its 125+ members, global technology providers, academics, and researchers, MLCommons is focused on collaborative engineering work that builds tools for the entire machine learning industry through benchmarks and metrics, public datasets, and best practices.

For additional information on MLCommons and details on becoming a member or affiliate, please visit MLCommons.org or email [email protected].