MLPerf Client
Create an application that contains a set of fair and representative machine-learning benchmarks for client consumer systems.
Purpose
The goal of the MLPerf Client working group is to produce machine-learning benchmarks for client systems such as desktops, laptops, and workstations based on Microsoft Windows and other operating systems. The MLPerf suite of benchmarks is the gold standard for AI benchmarks in the data center, and we are bringing our collaborative, community-focused development approach and deep technical understanding of machine learning (ML) to creating a consumer client systems benchmark suite.
The MLPerf Client benchmarks will be scenario-driven, focusing on real end-user use cases and grounded in feedback from the community. The first benchmark will focus on a large language model, specifically, the Llama 2 LLM.
Deliverables
- Client benchmark rules and definitions
- Client benchmark reference software
- Client benchmark submission rules
- Client benchmark roadmap
- Client benchmark app for Windows and Mac (future version)
- Development of our first large language model benchmark
Meeting Schedule
Wednesday December 11, 2024 Weekly – 09:05 – 10:00 Pacific Time
How to Join and Access MLPerf Client Working Group Resources
The MLPerf Client working group is limited exclusively to MLCommons members and affiliates. If you are not already a member or affiliate or part of a member or affiliate company, you can learn more about MLCommons Membership here.
To sign up for the group mailing list, receive the meeting invite, and access shared documents and meeting minutes:
- Fill out our subscription form and indicate that you’d like to join the Client Working Group.
- Associate a Google account with your organizational email address.
- Once your request to join the Client Working Group is approved, you’ll be able to access the Client folder in the Members Google Drive.
To engage in working group discussions join the group’s channels on the MLCommons Discord server.
GitHub repositories are still under construction.
Client Working Group Chairs
To contact all MLPerf Client working group chairs email [email protected].
Jani Joki
Jani is a Director of Benchmarking at NVIDIA and handles a variety of external and internal benchmarking efforts across all of NVIDIA. He has 25 years of experience in creating benchmarks, most notably having led the creation of such industry-standard benchmarks as 3DMark and PCMark in his previous position. Jani holds a Bachelor’s of Computer Science degree from Espoo-Vantaa Institute of Technology.
Ramesh Jaladi
Ramesh Jaladi is a Senior Director of Engineering in the IP Performance group at Intel responsible for performance analysis and optimization of client and datacenter technologies and products. In his 20+ years of performance engineering-focused career, he has helped develop industry metrics for and optimize the performance of client and server CPUs, hyperconverged infrastructure, video codecs, camera quality, and browser runtimes. Ramesh holds a MS in computer engineering from WVU and an MBA from Santa Clara University.
Yannis Minadakis
Yanni leads an organization at Microsoft in Windows+Devices focused on 3D Graphics and ML workloads on Windows. His passion is working on stream compute problems in both client and cloud. Yanni has worked at Microsoft for over six years. He has also worked at industry-leading companies such as Apple, Intel, Google, AMD, and Bloomberg. Yanni works with multiple startups, such as InterviewKickstart, to help engineers grow across all disciplines. He completed his master’s and bachelor’s degrees at Boston University.
