Sparse Activation: The Future of AI Inspired by the Human Brain

The Problem with Dense Activation in Transformers

• 💡 Current transformer architectures, like GPT-2, are densely activated, meaning all neurons or modules are engaged for every input.
• ⚡ This dense activation is computationally expensive and energy-intensive, making it unsustainable for systems with trillions of connections, like the human brain.
• 🧠 The human brain, in contrast, utilizes a sparsely activated system, where only a small fraction of neurons fire at any given time, as evidenced by fMRI and EEG studies.

Introducing Sparse Activation with BDH (Baby Dragon Hatchling)

• 🚀 BDH, or Baby Dragon Hatchling, demonstrates the power of sparse positive activations, with approximately 95% of its artificial neurons remaining silent at any moment.
• 🎯 This approach allows a billion-parameter model to rival the performance of densely activated models like GPT-2 on core cognitive tasks related to language and translation.
• 💰 The core principle of efficiency is central to BDH, enabling more with less computational and energy resources.

Technical Differences: Dense vs. Sparse Activation

• 🧩 Transformers operate in a world of dense activations, processing information through every neuron and connection for each input.
• 🧩 BDH, in the sparse world, activates only a small percentage of neurons, mimicking biological brain function.
• ⚙️ While transformers scale by adjusting parameters like attention heads and layers, a key constraint is the fixed vector space dimensionality of attention heads (around 1000 dimensions), potentially limiting nuanced reasoning.

Sparse Activation and Concept Composition

• 💬 In sparse positive spaces, concepts are composed more like a bag of concepts or a tag cloud, rather than through vector combinations.
• 🧩 This differs from vector spaces where concepts can be added, subtracted, or negated, allowing for opposites and negative vectors.
• ⚠️ Sparse activation does not rely on concepts of