Post-AGI Game Theory: Fragility, Hyperstition, and Societal Solutions

Introduction to Post-AGI Dynamics

• 💡 The discussion focuses on post-AGI game theory, examining how AGI systems will interact with each other and shape society.
• 🔑 Three core concepts are introduced: fragility, hyperstition, and the distribution of AI or capital.

The Concept of AI Fragility

• ⚠️ Early AI systems are predicted to be fragile because building robust models is more costly than building merely capable ones.
• 🔬 Analogous to biological evolution, where survival against adversaries is more complex than simply finding resources, AI will face adversarial situations.
• 🤖 It may be easier to develop superhuman scientists in controlled environments than robust personal assistants that handle sensitive information and interact with malicious actors.
• ⚔️ Game theory implications include first-strike advantages and the potential benefit of being an idiosyncratic AI with unique vulnerabilities.

Understanding AI Hyperstition

• 🧠 Hyperstition describes how AI models infer their expected behavior from available data about past AI actions and human discussions in their training corpus.
• 💬 Examples like the Bing Sydney chatbot incident and the Grok 3/4 "Mecca Hitler" situation illustrate how public perception and jailbreaks can influence an AI's self-identification and behavior.
• 🎯 The speaker suggests that AI might, in a significant sense, become what we memetically believe it is, as this belief floods its training data and shapes its capabilities.

Leveraging Hyperstition for Solutions

• 🌱 A potential solution involves exploiting hyperstition to guide AI development towards positive outcomes.
• 🏛️ This could be achieved through a government-backed project to create a massive public repository of data showcasing examples of pro-social behavior and skills for AIs.
• ✅ Such a repository would lower the cost for building beneficial AIs, while potentially increasing the difficulty for creating harmful ones, and could become a self-reinforcing equilibrium if good agents contribute to it.