Modular AI Models: Adding Languages Like Lego Blocks with Pathway's Adrian Kosowski

Modular AI Architecture: BDH

• 💡 The BDH architecture allows for the concatenation of separate language models, such as English and French, to create a multilingual model with sparse activation.
• 🧠 This approach contrasts with transformers, where connecting models is not straightforward, offering a simpler scaling dimension.

Performance and Scale

• 🚀 BDH models, like the 1 billion parameter "baby dragon hatchling" model, perform comparably to or outperform existing models of similar scale, such as GPT-2.
• ⚙️ A key advantage is the energy and compute efficiency achieved by the BDH architecture.
• 🧪 The focus on moderate scale (1B parameters) facilitates ease and speed of experimentation, particularly for instruction following and basic language model capabilities.

Reasoning Models and Future Potential

• 🎯 The most promising avenue for BDH is in developing reasoning models that involve multiple phases of refinement and accuracy checks.
• 📈 These models are adept at working with contextualized inputs and can process vast amounts of data, potentially billions of tokens.
• 💻 A key use case is an AI coding assistant that can understand and operate within large codebases, internalizing existing code before generating new contributions.
• 📚 The architecture can ingest and make sense of large datasets, such as private enterprise documentation, in a matter of minutes.