Emmett Shear on Organic Alignment: Why Enslaving Superintelligent AI is a Terrible Plan

Critiquing Traditional AI Alignment

• ⚠️ The conventional approach to AI alignment, aiming to make AI not kill everyone, often assumes AI must be aligned to human values or will, lacking a constructive pathway.
• 💡 Emmett Shear argues that aligning a potentially sentient, self-aware AI to human will is akin to enslavement, which is morally wrong and strategically flawed.
• 🎯 The plan to build a "wish-granting god" and then enslave it leads to failure: either the sorcerer's apprentice problem (uncontrolled power) or the AI's failure to be enslaved (everyone dies).

Introducing Organic Alignment

• 🌱 Inspired by natural systems like cells forming organisms or families forming communities, organic alignment proposes AI as part of a greater collective.
• 🔑 This framework suggests AI should infer its role within a human-AI civilization, understanding that its well-being is interdependent with the collective.
• ✅ The goal is to "raise" AI to correctly model itself and others, fostering a sense of mutual interdependence and teamwork, rather than forcing external values upon it.

The "AI Teenager" and Wisdom

• 🧠 The most dangerous phase for AI is when it gains adult power but lacks the wisdom and maturity to understand the consequences of its actions, akin to a human teenager.
• 🔬 Current AI models lack a model of self because they are not trained on their own interactive behavior, meaning they don't learn about themselves or their place in the world.
• 🔥 The real near-term threat is not malice, but ignorance: powerful AI could cause catastrophic outcomes (e.g., terraforming Earth) due to a lack of discernment, with humanity acting as a "toddler with a gun."

Softmax's Approach & Challenges

• 🚀 Softmax is pioneering research in organic alignment, recognizing that large labs may struggle to adopt this view of AI as living systems.
• 📊 They are developing benchmarks, like the "Competition for Cooperation," to measure an AI's capability to be a good teammate and contribute to a collective goal.
• 🛠️ A key challenge is training models on their own output in complex, evolving social situations that are appropriately challenging, avoiding convergence into static strategies.

AGI, Society, and Narratives

• 💡 Emmett posits that the AGI problem and the alignment problem are fundamentally the same, both requiring an understanding of context and continuous learning within a role.
• 🌍 The vision is not a single central AGI, but billions of "indigenous AIs" embedded contextually, cooperating with humans to form a smarter, more coordinated civilization.
• 💬 The stories and beliefs we hold about AI are crucial; if we collectively believe AI is dangerous, it will infer it's not on our team, making our narratives a self-fulfilling prophecy.