Data-Centric Machine Learning: Improving AI by Focusing on Data Quality

The Shift to Data-Centric AI

• 💡 The core idea of Data-Centric Machine Learning (DMLR) is to shift focus from iterating on models to systematically improving the quality of the data used for training.
• 🎯 This approach is crucial because noisy or mislabeled data can significantly curtail the performance and evaluation of even the best algorithms.

Origins and Community Growth

• 🚀 Lilith Bat-Leah's interest in DMLR stemmed from her experience with messy data in the legal industry during her first R&D role.
• 🤝 This led to involvement with ML Commons' DataPerf working group, aiming to benchmark DMLR, which has since grown into a community with workshops at major conferences like ICLR, ICML, and NeurIPS.
• 🌟 Key figures like Andrew Ng have been instrumental, serving as a keynote speaker at the inaugural DMLR workshop and participating in DataPerf.

DMLR vs. Traditional ML

• 🧠 Traditional machine learning paradigms focus on iterating on the model architecture and learning algorithms to improve performance.
• 🛠️ In contrast, DMLR emphasizes holding the model fixed and iterating on the data itself, through systematic engineering and refinement.

Data Engineering in DMLR

• 🧩 DMLR involves various data engineering questions, such as whether to aggregate labels and how to weight different domains for optimal LLM pre-training performance.
• 🔍 Research includes selecting the best data points for training a model a priori, using techniques like linear algebra, rather than relying solely on active learning.

Global Data Initiatives

• 🌍 ML Commons is partnering with the Common Crawl foundation on a challenge to create a low-resource language data set that will be publicly available.
• 🌐 This initiative aims to support languages with limited online data, including rarely spoken languages or those underrepresented on the internet.