Over the last eight years, we have seen tremendous growth and change within the AI industry, and here at MLCommons.  None of that growth would have been possible without you all, our members.

Today, we are updating and clarifying the MLCommons mission to reflect what has always been our core focus: 

Standardize AI measurement and enable informed decisions.

Measurement is fundamental to responsibly unlocking the tremendous potential of AI. These systems are the ultimate black boxes: they use intricate patterns across billions of numbers to generate sophisticated behavior. We have only vague, unsatisfying explanations of how AI systems work, and they are becoming more complex every day. Not only that, but training and operating AI systems is expensive and power-hungry. Better measurements enable us to empirically characterize the behavior of AI systems and their operational costs. 

If a characteristic of AI hardware, software, or data matters to the value delivered – be it speed, power, reliability, or quality – we aim to measure it. In doing so, we support informed decisions regarding the AI systems available today, enable cost-benefit tradeoffs, provide a north star for innovation, and support risk management such as assurance and insurance. Our work touches and benefits everyone involved in the AI industry – academic researchers, hardware and software vendors, deployers, end users, and even policymakers. 

We do this because it de-risks and grows the AI market.

We specifically aim to develop and operate standardized measurements in the form of industrial-quality benchmarks, which combine detailed protocols with artifacts such as test harnesses, held-out test data sets, and evaluation models. Given the complexity and (in many cases) subjectivity of AI testing, comparable measurements cannot be achieved by multiple implementations built according to a paper standard. Further, many of the artifacts must be technically and legally protected to avoid overfitting, and the measurements must evolve to keep pace with the technology – all of which requires a trusted, sustainably operating organization. 

We exist to meet these independent measurement needs and, in doing so, provide an essential service that supports the broader work of standards organizations such as ISO, Appia, CoSAI, and ADA.

Our work is to define, build, operate, certify, explain, evolve, and connect our benchmarks. Each of these activities is vital to our mission:

  • Defining what is being measured, be it an AI system under test or a dataset, is crucial to making our measurements standardized, understandable, and repeatable. Defining the metrics to be measured requires translating intuitions into rules for protocols, harnesses, expected behaviors, measurements, and result presentation. Throughout, we focus on technically rigorous definitions, developed through a transparent community process, to measure real value.  We do not engage in pay-for-play “benchmarketing”.
  • Building a benchmark is an exercise in careful engineering and, often, novel science. We need to translate messy, real-world use cases into sufficiently rigorous test scenarios as faithfully as possible. For subjective benchmarks, we need maintainable AI-driven evaluations that keep pace with AI’s progress and continue to be a valid measurement of the task of interest.
  • Operating a benchmark translates a GitHub repo and dataset into a living instrument. By providing operational infrastructure such as Candela and federated evaluation, we make our benchmarks both easy to use and high-integrity. 
  • Validating benchmark results provides third-party confirmation for performance claims, supporting purchasing, assurance, insurance, and even regulatory compliance. We aim to be the gold standard for accurate measurements, and to faithfully support other organizations that define requirements based on those measurements.
  • Explaining a benchmark is fundamental to making sure the results can be understood and used. We provide technical commentary on the issues underlying the benchmark. We bring together and brief experts who consume our results. We partner with standards and assurance organizations to incorporate the results into protocols and policies. 
  • Evolving a benchmark closes the loop. We seek to understand how our measurements are used and how they can be improved. We track improving technology and we monitor intensifying threats – then we use that information to ensure our benchmarks meet the moment. A benchmark is only valuable if it helps show the differences in tested systems, so we will need to continuously improve our work as systems get better.  
  • Connecting our benchmarks to other standards organizations and emerging policy frameworks globally creates a mature industry ecosystem that supports our members in delivering ever increasing value to their customers with lower risk and higher efficiency.

In the coming days and months, we will make organizational changes to better support this vital mission. These changes will include:

  • Better serving consumers of results by improving our website and providing more supporting resources to make it easier for them to find and use our results, such as we are doing with MLPerf Endpoints.
  • Promoting our benchmarks in new ways by bringing more product and marketing thinking onto our board and staff and building out the Candela service.
  • Enabling efficiency tradeoffs by aligning our work on performance and reliability at both an organizational and technical infrastructure level. We will support AI vendors and deployers in delivering increasing quality at lower cost and power consumption.
  • Broadening our community by encouraging the AI deployers who are our members’ customers today to join as members. To support these new members, we will refactor our fee structure to scale costs more proportionately with benefits.
  • Growing our scope to serve more people by exploring benchmarking for critical markets such as efficiency, healthcare, and financial services.

We welcome your ideas and feedback as we undertake these changes. We will continue to evolve as the field progresses, and new areas such as AI data science advance. MLCommons is your organization, and we aim to deliver maximum value to you, our members. 

What we do would not be possible without your involvement and support. Thank you. We look forward to partnering with you to measure AI and inform the growth of this amazing industry and its thoughtful adoption by society.

Kind regards,

Peter