Best Practices Working Group
Benchmark Infra Working Group
- Overview
- Training Working Group
- Inference Working Group
- Datasets Working Group
- Best Practices Working Group
- Research Working Group
Mission
Make machine learning more reproducible and easier to manage for the broader community by building logging tools and recommending approaches for tracking and operating machine learning systems.
Purpose
Across MLCommons® projects, we strive to simplify user experience by providing a unified set of tools. Centralized logging tools are especially critical because they simplify rules compliance and ensure that all vendor submissions for MLPerf™ benchmarks are easy to debug and capture the relevant ML system details.
This WG strives to improve reproducibility of results and automation of documentation about results. By understanding system-level specs and increasing reproducibility, we can start to build a more detailed matrix of performance-impacting factors. By improving automation, we can improve user experience and verify that each vendor submission includes requisite information.
Deliverables
- Logging and reporting tools for MLCommons projects
- Logging metrics and format
- Definition and examples of system specs
- Roadmap for unified logging tools across MLCommons projects aligned with inference, training, best practices, etc. roadmaps
- Best practices for MLPerf training and inference result reproducibility
Meeting Schedule
Weekly on Tuesday from 10:35-11:00AM Pacific.
How to Join
Use this link to request to join the group/mailing list, and receive the meeting invite:
Benchmark-Infra Google Group.
Requests are manually reviewed, so please be patient.
Working Group Resources
-
Shared documents and meeting minutes:
- Associate a Google account with your e-mail address.
- Ask to join our Public Google Group.
- Ask to join our Members Google Group.
- Once approved, go to the Benchmark-Infra folder in the Members Google Drive.
Working Group Chairs
Xinyuan Huang (xyhuang@mlcommons.org) - LinkedIn
Xinyuan is a technical leader at Cisco who is focusing on systems for ML ops and performance on both cloud and edge. Previously, he has also worked on cloud infrastructure optimizations and machine data analytics. He holds a Master's degree in Machine Learning from University College London, and Bachelor's degree from Fudan University.
Kongtao Chen (kongtao@mlcommons.org) - LinkedIn
Kongtao is a software engineer at Google, working on machine learning efficiency. He worked at Amazon for MXNet, a deep learning framework. He got his Ph.D. from University of Pennsylvania, Master's degree from Wharton Business School, and Bachelor's degree from Nanjing University.