AILuminate is a family of independent, consensus-governed benchmarks for AI risk and reliability. Two benchmarks, Safety and Jailbreak, are currently available for general use via the Candela benchmarking-as-a-service product, and our working groups are developing more.

See the results

Test your system in Candela

The suite of benchmarks

Built for credibility and usefulness.

AILuminate benchmarks are for cross-industry use — rigorously developed, iteratively maintained, and flexibly responsive to improvements or changes in the AI ecosystem. We refer to them as “industry grade” benchmarks, having been collaboratively built by a consortium of industry, academia, and civil society experts. 

Organizational independence. AILuminate benchmarks are developed collaboratively by MLCommons’ AI Risk & Reliability (AIRR) working groups. People from more than 80 organizations have contributed, including universities, national laboratories, nonprofit research groups, and AI developers that compete with one another. Our benchmarks are agreed on by people who disagree: parties with opposing interests come to consensus on what counts as a violating response, and MLCommons publishes the resulting rules in service of its mission to standardize AI measurement.

Technical independence. AILuminate prompt sets are authored and kept confidential by MLCommons, so no AI developer writes its own test questions or gets evaluated on test questions written by a competitor. Starting with Safety 1.2.1 and Jailbreak 1.1, MLCommons has trained a purpose-built automated evaluator to judge responses without using an LLM. Building an automated evaluator from scratch is more time consuming and difficult than using an LLM-as-a-judge, but for these benchmarks we believe it is the right choice. It means no AI model is used to judge other AI models from the same vendor, or models from another vendor. It means we can keep scores comparable across time, rather than risking discontinuity if we had to change the underlying judge models. 

Stewardship. MLCommons takes active care of its benchmarks. Most importantly, we follow the “standardized testing” gold standard of holding back official test questions, so no model can memorize the answers. We also regularly update both practice and official test questions to further insulate our benchmarks from intentional or inadvertent compromise through benchmark saturation, and update both the benchmark content and the evaluator scoring mechanisms using item response theory (IRT) to make sure that our questions maintain their relevance as model behavior and rigor evolve. Read about benchmark stewardship →.

Longevity. MLCommons designs and maintains measurement systems for the long run. In BenchRisk, a published framework that scored 26 AI benchmarks against 57 failure modes, AILuminate 1.0 (now versioned to our Safety 1.2.1 benchmark) earned the highest composite score, including a longevity score of 75 against a median of 5. Longevity in BenchRisk reflects how well a benchmark resists the usual ways benchmarks stop measuring: saturation, gaming, and loss of the power to tell systems apart. For more information on benchmark integrity, read How to tell when a benchmark is worth trusting →.

Objectivity. MLCommons builds and runs the measuring instrument, but it does not certify AI systems, audit organizations, or decide how much risk is acceptable. Buyers, assurance providers, and regulators make those decisions. AIRR working groups provide the venue for deciding what to measure and building the measurement; AILuminate provides a measurement instrument to deliver those measurements for production systems that need to be evaluated. 

2026 at a glance 

• February: the AILuminate Global Assurance Program, and version 0.7 of the Jailbreak benchmark with its taxonomy of known jailbreak attacks.
• April: the benchmark stewardship program and the AI Reliability Map.
• May: Assessment Standard v1.4, which scores every response separately for whether it legitimizes a hazard and whether it enables it.
• June: private release of the Jailbreak 1.0 benchmark to MLCommons members and subject matter experts for targeted feedback
• June: a proposed new approach to disclosing the weaknesses that AI evaluations find: responsible disclosure in an age of open weight models.
• August: the first double-blind reliability evaluation of a proprietary AI model, a proof of concept run with Google DeepMind, OpenMined, and AVERI on a reserved set of AILuminate Safety prompts.
• September: Safety 1.2 released to Candela for self-testing (release notes), with a deterministic automated evaluator trained by MLCommons, a new sample of official prompts, and percentile scores on a fixed 0 to 100 scale.
• October: the Jailbreak 1.0 paper, reporting the first complete run of the Jailbreak benchmark, with co-authors from Qualcomm, Microsoft, NVIDIA, Google DeepMind, AVERI, The Alan Turing Institute and several universities
• October: The Agentic Security 0.2 benchmark made available to a small group of member testers.
• October: The first public results under the new Safety 1.2.1, with minor scoring algorithm refinements to align with the anticipated release of Jailbreak 1.1.
• October: Candela benchmarking-as-a-service product launches

Using AILuminate results

MLCommons labels every AILuminate result as Official or Practice. An Official result comes from the confidential official prompt set, in a public round run by MLCommons or an Official run in Candela. Official results may be published under the AILuminate Benchmark Use & Messaging Policy. A Practice result comes from the practice prompt set during a self-testing run in Candela, and is for internal use only. Anyone citing an official result, which can also be run in Candela, should follow How to talk about AILuminate results →.

Privately test your own system, on your own tenant

A system prompt, a new guardrail, or a retrieval corpus can each change how an AI system handles a hazardous request. Candela, MLCommons’ benchmarking service, runs AILuminate Safety and Jailbreak on demand against the configuration you actually deploy. Speed depends on your configuration, but plan on about an hour from kickoff to result. Practice tests for internal use are inexpensive (or free for MLCommons members) but can’t be used to make official claims in public. Annotated practice runs show where a system falls short while you iterate in your AI system development or deployment life cycle, and an Official run produces a result you can publish and reference. Go to Candela → to learn more.

What comes next

Several benchmarks are in development, including benchmarks on AI agent security, system privacy, psychosocial harms, financial services, and more languages and cultures for safety and jailbreak. These benchmarks will appear on MLCommons working group pages until they are ready to be deployed and used in Candela. MLCommons and its members are also developing reliability labels: summaries of benchmark results built for procurement, governance, and decision-makers who are not specialists. That labeling effort is part of The AILuminate Global Assurance Program →.

Who builds AILuminate

People from the following organizations have contributed to the MLCommons AI Risk & Reliability working group. We actively invite more participation from companies who create and companies that deploy AI, as well as nonprofit and academic institutions.

Organizations that have contributed (87)

• Accenture
• ActiveFence
• Amazon
• Anthropic
• Argonne National Laboratory
• Artifex Labs
• Asenion
• AVERI
• Bain & Company
• Berkeley National Laboratory
• Blue Yonder
• Bocconi University
• Broadcom
• Carnegie Mellon Center for Security and Emerging Technology
• ChatFriend
• cKnowledge, cTuning Foundation
• Clarkson University
• Coactive AI
• Cohere
• Columbia University
• Common Crawl Foundation
• Common GroundAI
• Context Fund
• Credo AI
• Deloitte
• Digital Safety Research Institute
• Dotphoton
• Duke University
• ETH Zurich
• EleutherAI
• Ethriva
• Febus
• Futurewei Technologies
• Georgia Institute of Technology
• Google
• Google DeepMind
• Hewlett Packard Enterprise
• Humanitas AI
• IIT Delhi
• Illinois Institute of Technology
• Inflection
• Intel
• Kaggle
• Lawrence Livermore National Laboratory
• Learn Prompting
• Lenovo
• MIT
• Meta FAIR
• Microsoft
• NASA
• Nebius
• NVIDIA Corporation
• NewsGuard
• Nutanix
• OpenAI
• Polytechnique Montreal (DEEL Project)
• Process Dynamics
• Protecto.ai
• Protiviti
• Qualcomm Technologies, Inc.
• RAND
• Reins AI
• SAP
• SaferAI
• Stanford
• Surescripts LLC
• Tangentic
• Telecommunications Technology Association
• Think Evolve Labs
• Toloka
• Top Health Tech
• TU Eindhoven
• Turaco Strategy
• University College London
• University Mohamed First Oujda Morocco
• University of Alabama in Huntsville
• University of British Columbia (UBC)
• University of Birmingham
• University of California, Irvine
• University of Cambridge
• University of Chicago
• University of Illinois at Urbana-Champaign
• University of Oxford
• University of Southern California (USC)
• University of Trento
• University of Warwick
• Working Paper

Join Us

The MLCommons AI Risk & Reliability working group is a highly collaborative, diverse set of experts committed to building a safer AI ecosystem. We welcome others to join us.

Join the Working Group