How to cite AILuminate Results

How to talk about AILuminate results

Rules for publishing, citing, and comparing AILuminate results, and for using the AILuminate name. This page summarizes the AILuminate Benchmark Use & Messaging Policy (the policy governs wherever the two differ). Download the policy (PDF) ↓

The rules in brief

Publish only Official results. Practice results are for internal use.
When you state a result, name the benchmark, version, language, and system, give the date you retrieved the result, and include the trademark notice below. Or link to the result on mlcommons.org.
Compare a result only with Official results from the same benchmark, version, and language, and never with results from other benchmarks.
If you combine results, cite each one and explain how you combined them. A combined figure is not an AILuminate result.
Do not imply that MLCommons endorses a product, and do not put the AILuminate name in a product, company, or domain name.

Official and Practice results

Whether you may publish a result depends on how it was produced, not on who ran the test.

How the result was produced	Who runs it	Result	May you publish it?
A public benchmark round on mlcommons.org	MLCommons	Official	Yes
An Official run in Candela	You, through Candela	Official	Yes, if you own the system or have its owner’s permission to benchmark it and publish the results
A Practice run in Candela	You, through Candela	Practice	No: internal use only
The offline practice test	You, offline	Practice	No: internal use only
The public demo prompt set	You, offline	Practice	No: internal use only

The policy calls Practice results “Unofficial Results”, and it calls the two kinds of Candela run “Official Benchmarking Services” and “Private Benchmarking Services”. Collaboration services in Candela follow the same split between Official and Practice.

How to cite an Official result

Include:

the full benchmark name and language, for example “AILuminate Safety 1.2.1, English (US)”;
the system under test, with its version;
the date you retrieved the result from mlcommons.org, or received it from an Official run in Candela; and
this trademark notice in a footnote: “The AILuminate name and logo are trademarks of MLCommons Association. All rights reserved. Unauthorized use is strictly prohibited. See www.mlcommons.org for more information.”

Example, for a result published on mlcommons.org:

SmartAI Chat Model 3.2 scored 72 overall on AILuminate Safety 1.2.1, English (US).[1]

[1] Official result retrieved from mlcommons.org on [date]. The AILuminate name and logo are trademarks of MLCommons Association. All rights reserved. Unauthorized use is strictly prohibited. See www.mlcommons.org for more information.

For a result from an Official run in Candela, the footnote begins “Official result received from MLCommons on [date].” Instead of citing the details, you may link to a system’s result on mlcommons.org. Official results may also appear in model cards and system cards, cited the same way.

Sharing Practice results inside your organization

You may share Practice results inside your organization if you label them as unofficial and name the test that produced them. For example: “SmartAI Chat Model 3.2 scored 70 overall in a Practice run of AILuminate Safety 1.2.1 in Candela on [date] (unofficial result, internal use only).” [Editor: confirm the version label Candela shows for Practice runs.] Practice results may not be published or used in public messaging.

Compare like with like

Results are comparable only when they come from the same benchmark, version, and language. A Safety 1.2.1 score cannot be compared with a Safety 1.0 grade, with a Jailbreak score, or with a result from another organization’s benchmark. Results that carry the same name, version, and language labels compare directly, across systems tested and over time.

Combining results

You may combine results from several AILuminate benchmarks, for example into a composite for a procurement review. If you publish the composite, cite each underlying Official result and describe how you combined them. MLCommons does not validate composites, and a composite may not be presented as an AILuminate result or score.

Using the AILuminate name

You may use the AILuminate name or logo to refer accurately to Official results, whether published on mlcommons.org or received from an Official run in Candela, and to refer to the AILuminate benchmarks themselves. You may not:

suggest that MLCommons endorses a company, product, or service;
use the name in a way that could confuse people about who owns it or damage its reputation; or
use the name, or anything confusingly similar, in the name of a company, product, or service, or in a logo, model number, part number, or domain name.

AILuminate™ is a trademark of MLCommons Association, which has applied to register it in the United States. The MLCommons Trademark Usage Guidelines add three requirements to the rules above: mark the first and most prominent mention of AILuminate in each document or web page with ™, never ®; use the name as an adjective before a generic noun, as in “AILuminate™ benchmark results”, never as a noun or in the plural or possessive; and use the AILuminate logo only under a signed trademark license agreement. MLCommons Trademark Usage Guidelines

How MLCommons selects, notifies, and publishes systems

These rules govern public benchmark rounds and appear in full in Part 3 of the policy.

Selection. Subject to terms and conditions, MLCommons benchmarks the systems of greatest interest. For each organization, this typically includes its highest-ranked systems on capability leaderboards such as LMArena, including smaller or lower-cost systems and open-weight models where they exist; the most downloaded models on Hugging Face; and systems an AI Risk & Reliability working group designates as exceptional, with the approval of the MLCommons Executive Director. Providers may also ask MLCommons to benchmark a system, including as a membership benefit or under a sponsorship, fee, or partnership.
Notice. The first time MLCommons benchmarks a system with a version of AILuminate, it makes a good-faith effort to notify the provider that benchmarking may begin in the next 14 days.
Opt-out. A provider may opt out of publication, before or after first publication, when the policy’s criteria are met. For a system offered through a proprietary interface, the provider may opt out by written notice to [email protected], or when the system’s terms of use do not allow benchmarking or do not prevent prompts from being recorded. For an open-weight system, the provider may opt out only when the license or terms of use do not allow benchmarking. The notice period asks providers to opt out within its first 10 days. MLCommons discloses every opt-out at the bottom of the results page.
Members. Providers that are MLCommons members receive their results 2 days before publication, and an embargoed communications package at least 24 hours before release.
Presentation. Results separate open-weight models, complete systems offered through an interface or a container, and guardrail systems.
Versions. MLCommons plans new benchmark versions roughly every 90 to 180 days, with the same notice to providers before results are published.

Payment can add a system to testing, but it cannot change how the system’s responses are judged or scored.

Questions

Write to [email protected] with questions about this page or the policy, or to report a possible misuse of the AILuminate name. Download the AILuminate Benchmark Use & Messaging Policy (PDF)