One Model To Learn Them All

Ambitious Goal: Unified Deep Learning

• 💡 The 2017 paper "One Model To Learn Them All" explored if a single deep learning model could perform well across diverse tasks like images, speech, and text simultaneously.
• 📌 Historically, deep learning was siloed, with specialized architectures (e.g., CNNs for vision, RNNs/LSTMs for language) requiring extensive, task-specific research and tuning.
• 🎯 While multitask learning existed, it was mostly within the same domain, making a truly cross-domain, competitive model a significant challenge.

Bridging Data Diversity: Modality Nets

• 🧠 The first innovation was Modality Nets, acting as intelligent translators at the model's edges to handle vastly different data types (pixels, waveforms, text tokens).
• 🚀 These nets transform raw data into a common, unified representation (a high-dimensional vector space) that the central model can process, and then translate it back for task-specific outputs.
• ✅ Modality nets are computationally minimal, variable-sized to avoid information bottlenecks, and shared across tasks of the same modality for efficiency and generalization.

Core Architecture: Mixed Building Blocks

• 🧩 The central model deliberately combined previously domain-specific computational blocks, including depthwise separable convolutions for local patterns and efficiency.
• 💡 Attention layers (multi-head dot-product attention with timing signals) were crucial for tasks like translation, enabling the model to focus on relevant input parts.
• ⚡ Sparsely Gated Mixture of Experts (MOE) layers allowed for massive model capacity (billions of parameters) while only activating a small subset of experts per input, optimizing computation.

Performance & Joint Training Benefits

• 📊 The single model was trained concurrently on eight large datasets (ImageNet, speech recognition, image captioning, parsing, machine translation pairs) from distinct domains.
• 📈 It achieved generally competitive performance across all tasks, often close to highly tuned specialized models, without requiring extensive task-specific adjustments.
• 🔥 A key finding was that joint training significantly improved data-scarce tasks; for example, parsing accuracy jumped from 11.7% (alone) to 14.5% when trained with all eight tasks, including vision data.

Implications for General Intelligence

• 🔍 The paper provided strong evidence for transfer learning across domains, where knowledge from data-rich tasks (like vision) benefited data-scarce ones (like language parsing).
• 💡 Ablation studies showed that integrating diverse blocks (convolutions, attention, MOE) was beneficial across the board, suggesting a synergy rather than detriment outside their "home" domains.
• 🧠 This work hinted at the existence of universal computational principles or "computational primitives" that are valuable for processing information regardless of input type, a foundational step towards general intelligence.