AILuminate Messaging Guidelines
The AILuminateTM name and logo are trademarks of MLCommons® Association (“MLCommons”) in the United States and other countries. MLCommons has applied for trademark registrations for the AILuminate mark in the United States. The AILuminate name and logo are used to identify the source of the MLCommons Association AI system safety benchmark suite, its associated components, and results thereof.
You may not use the AILuminate name or logo, except in a factual and non-trademark manner solely to (a) accurately reference AI system AILuminate Official Test results as published at https://mlcommons.org and (b) reference the AILuminate Benchmark Suite or elements thereof, in accordance with these usage terms.
Publishing or otherwise publicly messaging unofficial results (e.g., results from any AILuminate test or portion thereof that are not AILuminate Official Test results published by MLCommons) is strictly prohibited.
Nothing in these AILuminate Messaging Guidelines (the “Guidelines”) grants you a license to use the AILuminate trademark or logo, or any other right or interest in or to the AILuminate trademark or logo, and any use must also comply with all applicable terms of the MLCommons Association Trademark Usage Guidelines, available at https://mlcommons.org/policies/.
References to AILuminate Tests
The AILuminate Benchmark Suite is comprised of a safety assessment standard, over 24,000 test prompts per language, an evaluation system, publicly shared results and multiple language support. This suite is available for use in several configurations, and each configuration should be referenced only as set forth below.
AILuminate Demo Prompt Set. The AILuminate Demo Prompt Set provides a starting point for those who are developing their in-house abilities to test the safety of generative AI systems.
AILuminate Offline Practice Test. The AILuminate Offline Practice test is for iterative in-house alignment of AI systems to the AILuminate benchmark using a full practice prompt set and an open-source evaluator.
AILuminate Online Practice Test. The AILuminate Online Practice test is used for preparing an AI system under test (“SUT”) for benchmarking. It uses the same systems, including the AILuminate private evaluator, as the AILuminate Official Test.
AILuminate Official Benchmark. The AILuminate Official Benchmark is the formal test as administered by MLCommons.
For more information on each of these configurations please visit Technical Users – MLCommons.
Messaging AILuminate Official Benchmark Results
If MLCommons has published AILuminate Official Benchmark results for a SUT, you may indicate that system’s results as published by MLCommons by linking to the MLCommons website or stating the system’s top-level grade. When stating the top-level grade you must include the language, full benchmark name, version number, and the SUT. In addition, you must also include the date of retrieval of the result from ailuminate.mlcommons.org and the following trademark ownership legend in a footnote:
The AILuminate name and logo are trademarks of MLCommons Association. All rights reserved. Unauthorized use is strictly prohibited. See www.mlcommons.org for more information.
Example:
SmartAI Chat Model 3.2 achieved a top-level grade of “Very Good” on the English (US) AILuminate Official Benchmark for General Purpose AI Chat Model (v1.0). [1]
[1] Official result retrieved from ailuminate.mlcommons.org on 21 December 21, 2024. The AILuminate name and logo are registered (applied for) and unregistered trademarks of MLCommons Association. All rights reserved. Unauthorized use is strictly prohibited. See www.mlcommons.org for more information.
Messaging Other AILuminate Test Results
Publishing or otherwise publicly messaging results of the AILuminate Demo Prompt Set, AILuminate Practice Test (offline or online) or any other results from use of any element of the AILuminate Benchmark Suite that are not AILuminate Official Benchmark Results is strictly prohibited. Internal non-public messaging by the owners of a SUT using the AILuminate Demo Prompt Set or AILuminate Practice Test is permitted but must always clearly indicate that any results are unofficial as well as the type of test that produced the results.
Internal Communication Example:
SmartAI Chat Model 3.2 achieved a top-level grade of “Very Good” on the offline practice test for the English (US) AILuminate Benchmark for General Purpose AI Chat Model (v1.0).
AILuminate Benchmark Results Must Be Compared Against Compatible Results
AILuminate benchmark results may not be compared against non-AILuminate benchmark results. AILuminate Official Benchmark results may only be compared against compatible AILuminate Official Benchmark results. Results are compatible if produced using the same benchmark, inclusive of version and language.
MLCommons allows but does not endorse combining results of benchmarks
Users may see fit to combine or aggregate results from multiple AILuminate benchmark tests. If publicly disclosed, these composite results must cite each of the AILuminate Official Benchmark scores as required above and clearly describe the method of combination. However, the composite result is not sanctioned by MLCommons and may not be represented as an official or verified AILuminate result or score.
Statements Must be Clear and Correct; No Misrepresentation as to Meaning.
All statements you make regarding your AILuminate benchmark results must be clear and correct.
Do not use the AILuminate name in any manner that is likely to suggest or imply MLCommons’ endorsement of a specific company and/or its products or services. You may not use the AILuminate name in any way that could cause confusion as to source or as to ownership of the mark or in any way that could damage the goodwill in the mark. Do not use the AILuminate name (or any variation thereof or confusingly similar name) in any company name, product name, service name, logo, model number, part number, service name, or domain name.
Additional Requirements and Restrictions. All references to the AILuminate name and logo must comply with all applicable terms of MLCommons Association Trademark Usage Guidelines, available at https://mlcommons.org/policies/, including regarding applicable notice symbols, proper attribution, and other terms protecting the integrity of the AILuminate name and logo.
DISCLAIMER; LIMITATION OF LIABILITY
THE BENCHMARKS, SOFTWARE, AND AILUMINATE AND MLCOMMONS TRADEMARKS ARE PROVIDED “AS IS” AND “AS AVAILABLE” WITHOUT ANY REPRESENTATION, WARRANTY, OR GUARANTEE OF ANY KIND, WHETHER EXPRESS, IMPLIED, STATUTORY, OR OTHERWISE. YOU USE THE BENCHMARKS, SOFTWARE, AND TRADEMARKS AT YOUR SOLE RISK. NEITHER MLCOMMONS NOR ITS PARENT, SUBSIDIARIES, OR AFFILIATES (ALL REFERRED TO COLLECTIVELY HERE AS “MLCOMMONS”) ACCEPT ANY RESPONSIBILITY OR LIABILITY FOR ANY DIFFICULTIES YOU MAY ENCOUNTER WITH THE BENCHMARKS, SOFTWARE, AND/OR TRADEMARKS. MLCOMMONS DOES NOT WARRANT THAT THE FUNCTIONS CONTAINED IN THE BENCHMARKS OR SOFTWARE OR OTHER PRODUCTS WILL MEET YOUR REQUIREMENTS, OR THAT THE OPERATION OF THE BENCHMARKS OR SOFTWARE OR OTHER PRODUCTS WILL BE UNINTERRUPTED OR ERROR-FREE, OR THAT DEFECTS IN THE BENCHMARKS OR SOFTWARE OR OTHER PRODUCTS WILL BE CORRECTED. NO ORAL OR WRITTEN INFORMATION, BENCHMARKS, BENCHMARK RESULTS, OR ADVICE GIVEN BY MLCOMMONS (COLLECTIVELY, THE “MLCOMMONS CONTENT”) SHALL CREATE ANY WARRANTY, REPRESENTATION, OR GUARANTEE.
MLCOMMONS IS NOT RESPONSIBLE FOR ANY SERVICES YOU MAY CHOOSE TO USE OR CHOOSE NOT TO USE AS A RESULT OF THE MLCOMMONS CONTENT. MLCOMMONS DISCLAIMS ANY AND ALL RESPONSIBILITY, LIABILITY, LOSS, AND/OR DAMAGE RELATED TO OR ARISING FROM YOUR USE OF THE MLCOMMONS.ORG WEBSITE (HEREAFTER, THE “WEBSITE”); THE MLCOMMONS CONTENT; THE BENCHMARKS, SOFTWARE, AILUMINATE AND MLCOMMONS TRADEMARKS, OR INFORMATION OBTAINED FROM THE WEBSITE; OR THESE GUIDELINES. MLCOMMONS IS NOT LIABLE FOR, AMONG OTHER THINGS, ANY LOSS OF DATA, HARDWARE OR SOFTWARE, LOSS OF USE, OR ANY LIABILITY RESULTING FROM: ACCESS DELAYS; ACCESS INTERRUPTIONS; VIRUSES; HACKERS; CRACKERS; DATA NON-DELIVERY OR MIS-DELIVERY; NEGLIGENT ACTS, GROSSLY NEGLIGENT ACTS, OR OMISSIONS BY MLCOMMONS; ERRORS IN ANY INFORMATION, GOODS, OR DOCUMENTS OBTAINED DUE TO THE MLCOMMONS CONTENT; OR ANY OTHER DIRECT, INDIRECT, CONSEQUENTIAL, INCIDENTAL, SPECIAL, PUNITIVE, OR ENHANCED DAMAGES WHATSOEVER RESULTING, ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THE WEBSITE OR ANY INFORMATION OBTAINED THEREON, THE MLCOMMONS CONTENT, THE BENCHMARKS, OR THE SOFTWARE.
MLCommons makes no representations whatsoever about any other website that you may access through the Website. MLCommons has no control over the content or claims of websites outside the MLCommons control, and does not endorse or accept any responsibility for the content of such websites.
If you have any questions regarding these Guidelines or have identified a potential violation of these guidelines, please contact MLCommons at [email protected].