Data-centric ML

Purpose

The Data-centric ML Research (DMLR) working group challenges existing ML benchmarking dogma by driving novel approaches to benchmark curation such as dynamic adversarial data collection. Benchmarks for machine learning solutions based on static datasets have well-known issues: they saturate quickly, are susceptible to overfitting, contain exploitable annotator artifacts and have unclear or imperfect evaluation metrics. This new paradigm of data-centric benchmarking is powered by the Dynabench platform. The key scientific question we investigate in this working group is: is it possible to make faster progress if data is collected dynamically, with humans and models in the loop, rather than in the old-fashioned static way? This further enables an ecosystem of other ML benchmarks in areas such as ML datasets and algorithms for working with datasets. We leverage these benchmarks through challenges and leaderboards.

Deliverables

DataPerf

• Organize three workshops at major ML conferences per year
• Recruit more participants to WG
• Measure and improve quality of Common Crawl (based on human annotation)
• Domains for impact challenges
  • LLMs + Commoncrawl
  • Safety
  • Science
• Research challenges beyond the chosen domains

Dynabench

• Supporting existing tasks, add three new tasks a year
  • A major task, such as Safety
• Support academic research using Dynabench
• Product improvements for LLM experiments

Shared objectives:

• Come up with sustainable funding model for ongoing research
• Develop approach to product management of Dynabench
• Define standard processes and decision-making criteria around accepting new challenges
• Establish standard processes for promoting new challenges and driving engagement