Mobile Working Group

Mission

Create a set of fair and representative inference benchmarks for mobile consumer devices such as smartphones, tablets, and notebooks that is representative of the end user experience.

Purpose

The MLPerf™ Mobile working group aims to collaboratively develop a performance-accuracy benchmark suite for consumer mobile devices with different AI chips and software stacks. The MLPerf Mobile working group draws from the expertise of mobile SoC vendors, ML framework providers, and model producers, and extends the MLPerf inference group’s efforts to a mobile context. We welcome new members that hope to raise the bar of ML performance for mobile devices.

Deliverables

1. Mobile benchmark rules and definitions
2. Mobile benchmark reference software
3. Mobile benchmark submission rules
4. Mobile benchmark roadmap
5. Mobile benchmark app for Android and iOS (future version)
6. Publish mobile benchmark results every ~6 months

Meeting Schedule

Weekly on Wednesday from 3:05-3:45PM Pacific.

How to Join and Access Working Group Resources

This group is limited to exclusively Members and Affiliates. If you are not already a Member/Affiliate or part of a Member/Affiliate company, you can learn more about Membership here.

• To sign up for the group mailing list, receive the meeting invite, and access shared documents and meeting minutes:
  1. Associate a Google account with your organizational email address.
  2. Request to join the Mobile Google Group. Requests are manually reviewed, so please be patient.
  3. Once your request to join the Mobile Google Group is approved, you'll be able to access the Mobile folder in the Members Google Drive.
• To engage in group dicussions:
  1. Join the group's channels on the MLCommons Discord server.
• To access the GitHub repositories (public):
  1. If you want to contribute code, please sign our CLA first.
  2. Visit the GitHub repositories:
     • Mobile App Open
     • Mobile Models
     • Mobile Open

Working Group Chairs

To contact all Mobile working group chairs email mobile-chairs@mlcommons.org.

Koan-Sin Tan (freedomtan@mlcommons.org) - GitHub - Twitter

Koan-Sin Tan is a Senior Technical Manager with MediaTek Inc. He has worked on performance engineering for the past decade. He enjoys hacking together small utilities and contributing to some big open-source projects from time to time. He has a PhD in Information Management from Chiao-Tung University, Taiwan.

Mostafa El-Khamy (elkhamy@mlcommons.org) - LinkedIn - Google Scholar

Mostafa El-Khamy received a B.S. and M.S. in electrical engineering from Alexandria University, Egypt, a M.S. and Ph.D. in electrical engineering from the California Institute of Technology (Caltech), USA, and a M.B.A. from the Edinburgh Business School, U.K. He is currently a Senior Principal Engineer with Samsung Device Solutions Research America, CA, USA.