We are extremely excited to announce the release of the Dollar Street Dataset, an open access diverse computer vision (CV) dataset. It is designed to reduce geographic and socioeconomic bias and is a collaboration between MLCommons®, Coactive AI, Gapminder – a Swedish non-profit, and Harvard University. The Dollar Street Dataset includes 38K high-quality images of household objects from across the world labeled with object tags and demographic data. The images were collected from dozens of different countries and all metadata was manually verified. The dataset is available under CC-BY 4.0 for academic and commercial usage. We improved classification accuracy for items from lower income households by 50%, highlighting the impact of data to reduce bias and ultimately empowering the community to build better machine learning for everyone.
Computer vision is an incredibly powerful discipline within AI for enriching society and the human experience and in the last decade, machine learning has become the linchpin of CV. Advances in machine learning make life subtly better – for example, helping us take vivid photos, capturing beautiful and unique memories. Other applications are profoundly shaping society and saving lives by enabling cars to detect pedestrians or helping detect, diagnose, and treat cancers better.
However, most existing commercial and open access datasets for training ML models disproportionately skew towards the developed world and more specifically high-income populations. This systematic bias in datasets means that many ML models perform unevenly across the broader population. In particular, many CV tools perform poorly for women and people of color. Just as one example, studies have found that existing image classification tools perform >30X worse for darker-skinned women compared to lighter-skinned men. This variation in performance can have tremendous real-world consequences, given the rapid adoption of CV throughout modern society.
The Dollar Street Dataset is an open access, diverse CV dataset designed to combat bias and enable machine learning to work better for everyone. It comprises 38K high-quality images of 289 everyday objects from households across the world, labeled with object tags and demographic data such as region, country, and household monthly income. The images were collected at scale across 63 different countries, including many communities without internet access, and all metadata was manually verified.
We also demonstrated that the Dollar Street Dataset can profoundly improve machine learning models by dramatically reducing bias and improving robustness – especially for lower socioeconomic communities. We used five standard CV models trained on existing datasets to classify images of household objects such as stoves, toilets, and toothbrushes. These models achieved an acceptable 57% accuracy for high-income households, but dropped to 18% for lower-income households – illustrating the bias of models trained on existing datasets. By finetuning with the Dollar Street Dataset, we achieved 70% or better accuracy across all household income levels, an amazing increase of 50% or more.
Contributors to the Dollar Street Datasets include researchers from Coactive AI, Harvard University, and MLCommons. It can be downloaded at mlcommons.org/dollar-street and for more information, please read our paper accepted to the 2022 Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks.
