Scale Can’t Overcome Pragmatics: The Impact of Reporting Bias on Vision-Language Reasoning

The Challenge of Vision-Language Reasoning

• 💡 Vision-Language Models (VLMs), despite high performance on various benchmarks, still struggle with basic reasoning capabilities.
• 🎯 For example, models like Lava fail to correctly identify simple spatial relationships, such as a mug being under a table.
• 🔑 This persistent issue suggests a fundamental problem beyond just model architecture or pre-training tasks.

Understanding Reporting Bias

• 🧠 The core hypothesis is that VLM limitations stem from reporting bias in their training data, where human captioners omit tacit information.
• 📚 Drawing from linguistics and pragmatics (e.g., Gricean maxims like quantity, relevance, and manner), people naturally avoid stating obvious or redundant information.
• 🚫 This leads to a significant underrepresentation of spatial, temporal, counting, and negation language in web-scale image-text corpora.

Impact on Model Performance

• 📊 Analysis of datasets like Leyon confirms a very low occurrence of reasoning-related language, far less than common descriptive terms.
• 📉 Contrastive models (e.g., Clip) perform poorly on these reasoning tasks, often ignoring negations entirely, while generative models (e.g., Lava, Malmo) show improvement but still fall short of human accuracy.
• 📈 Unlike recognition tasks, scaling data or model parameters does not significantly improve reasoning capabilities; the human behaviors causing reporting bias persist regardless of scale.

The Solution: Intentional Data Curation

• ✅ A controlled study demonstrates that clear and specific annotator instructions can successfully elicit the missing reasoning-related information.
• 🛠️ By explicitly asking annotators to describe counts, positions, temporal events, and negations, the presence of these concepts in captions increases dramatically.
• 🚀 The amount of data collected through such intentional curation is sufficient for models to learn these underlying reasoning tasks, highlighting the critical role of data quality over sheer scale for VLM development.