Providing leadership and expertise towards developing standardized measurements for AI Safety.
MLCommons® is pleased to announce our membership in the National Institute of Standards and Technology (NIST) Artificial Intelligence Safety Institute Consortium. We are contributing to NIST’s efforts establishing a new measurement science that will enable the identification of proven, scalable, and interoperable measurements and methodologies to promote development of trustworthy Artificial Intelligence (AI) and its responsible use. Few organizations are better-suited to take a leadership role for developing standardized measurement approaches to understand the safety of AI systems than MLCommons, and we’re honored to join the Consortium.
Our engagement with the Consortium will be driven by the work of our AI Safety Working Group (AIS), which was formed last November. Though the AIS is only a few months old, we’ve already seen tremendous engagement and progress toward collaboratively building a shared AI Safety platform and set of benchmarks for large language models, using Stanford’s groundbreaking HELM framework. Our dedicated engineering team is working closely with members of three active workstreams: stakeholder engagement, platform technology, and benchmark and tests, who are building the technology platform on top of which benchmarks and tests can be run, and defining the initial benchmark and test suites. Our ambition is to have a prototype of the platform up and running in the first half of this year.
We’re humbled by the broad engagement and progress we’ve seen so far in the AIS. We were thrilled to have over 90 participants from 40 organizations present at our North America kick-off meeting in December, and recently held a European kick-off meeting at the Sorbonne Center for Artificial Intelligence in January. If you’d like to join the effort, visit the AIS Working Group page for more information on how to participate.
Please note that NIST does not evaluate commercial products under this Consortium and does not endorse any product or service used. Additional information on this Consortium can be found here.
These are the MLCommons comments to NIST.
