MLPerf Automotive
Define and develop an industry standard ML benchmark suite for automotive to be used in request for information (RFIs) and request for quotation (RFQs).
OEMs and automotive suppliers regularly send out RFIs and RFQs to vendors to understand a solution’s compute performance and system resource utilization. This is a crucial step when assessing whether the sourced part is a suitable choice when designing the next-generation automotive compute platforms.
Purpose
The benefit of having an industry AI automotive benchmark suite is two-fold: it makes it easier to do fair (so called apples-to-apples) comparisons between different technologies, and to help guide where to focus engineering efforts for developing future hardware and optimizing current software. These benchmarks will help drive the industry forwards faster.
Safety – AI/ML enhances safety features such as collision avoidance, lane departure warning and other intelligent driving features. Measuring AI performance ensures automotive systems meet specific latency and other performance KPIs.
Efficiency – AI is used to optimize vehicle performance, including fuel efficiency and predictive maintenance.
User Experience - Consumers have a better experience in a vehicle with smart functions.
Regulatory Compliance – Automotive regulation is very strict and collaboration is key to meeting regulations around the use of AI in vehicles, particularly in relation to safety. Industry driven alliances keep us ahead of regulatory requirements.
Deliverables
The Automotive task force will create a MLPerf benchmark suite for automotive running on systems developed for automotive purposes. The focus will be on, but not limited to, camera sensor perception as this is currently the most mature type of AI or automotive. However, the use of AI in new areas is increasing as more features adopt more sophisticated algorithms. With time, the benchmark suite will be updated to take these new use-cases into consideration.
Meeting Schedule
Wednesday December 18, 2024 Weekly – 08:05 – 09:00 Pacific Time
Why is AI/ML Important for Automotive designs?
Safety
AI/ML enhances safety features such as collision avoidance, lane departure warning, and other intelligent driving features. Measuring AI performance ensures automotive systems meet specific latency and other performance KPIs.
Efficiency
AI is used to optimize vehicle performance, including fuel efficiency and predictive maintenance.
User Experience
Consumers have a better experience in a vehicle with smart functions.
Regulatory Compliance
Automotive regulation is very strict and collaboration is key to meeting regulations around the use of AI in vehicles, particularly in relation to safety. Industry driven alliances keep us ahead of regulatory requirements.
Technical Approach
Measuring automotive AI performance requires focus on specific areas in the image processing chain. The following figure illustrates how the MLPerf Automotive benchmarking task force (ABTF) is focused strictly on PERCEPTION processing for the first benchmark demonstration.
The ABTF benchmark will leverage the existing MLCommons LoadGen infrastructure running on dedicated AI automotive hardware. The following diagram illustrates the high-level LoadGen control concept.
ABTF Projects
How to Join and Access MLPerf Automotive Resources
To sign up for the group mailing list, receive the meeting invite, and access shared documents and meeting minutes:
- Fill out our subscription form and indicate that you’d like to join the MLPerf Automotive task force.
- Associate a Google account with your organizational email address.
- Once your request to join the Automotive task force is approved, you’ll be able to access the Automotive folder in the Public Google Drive.
To engage in Automotive task force discussions, join the task force’s channels on the MLCommons Discord server.
To access the GitHub repository (private):
- If you want to contribute code, please submit your GitHub ID to our subscription form.
- Visit the GitHub repositories:
Why are MLCommons and AVCC Partnering on Automotive AI Performance?
- MLCommons has a proven track record of developing well adopted industry standard ML benchmark suites Edge, Datacenter and Mobile
- AVCC has a proven track record of publishing Technical Reports on how to measurement ML performance for automotive
- Inaccurate AI performance measurements increase financial, technical and safety risk
- The MLPerf Automotive benchmark task force (ABTF) is focused on defining the right AI/ML performance targets and specifications to ensure fairness, accuracy and broad industry adoption
Automotive Working Group Task Force Chairs
To contact all Automotive Benchmark task force chairs email [email protected].
James Goel
James Goel leads the AI/ML standards group at Qualcomm where he is a Senior Director of Technical Standards. He is on the board of the Video Electronics Standards Association (VESA) and is the chair of the MIPI Alliance Technical Steering Group. Previously, he was the VP of Engineering at Silicon Optix before it was acquired by Qualcomm in 2011. He holds a Bachelor’s of Applied Science (B.A.Sc) in Electrical Engineering from the University of Waterloo in Canada and is a licensed Professional Engineer (P.Eng.) in Ontario where he’s practiced for the last 31 years.
Kasper Mecklenburg
Kasper Mecklenburg, based in San Jose, is a performance analysis engineer at Arm within automotive with focus on software-defined vehicles and on enabling AD/ADAS applications. Kasper has a Bachelor’s in Numerical Analysis and a Master’s in Applied Physics from Lund Institute of Technology, Sweden.
