Harlan Stewart & Liron Shapira: Critiquing Dario Amodei's AI Risk Understatement

Critique of Amodei's AI Risk Assessment

• 💡 Harlan Stewart's rebuttal addresses Dario Amodei's "The Adolescence of Technology," arguing that Amodei dramatically understates extinction-level AI risks.
• 🎯 Liron Shapira notes that Amodei's essay lacks a sense of alarm, despite acknowledging risks only indirectly.
• ⚠️ Stewart, from MIRI, assigns a 75% personal probability of catastrophic failure if current AI techniques scale unchecked.

Misrepresenting AI Risk Critics

• 🧠 Amodei's essay caricatures critics as fatalists who believe doom is inevitable, which is seen as backward.
• 💬 Critics are actively mobilized because they believe course correction is still possible, not because they've given up.
• 🚫 Harlan objects to Amodei's use of straw manning and character assassination, dismissing critics as religious or too theoretical, which leads to low-quality discourse.

Theory vs. Empirical Advantage

• 🔬 Amodei claims lab insiders have an empirical advantage, dismissing abstract theory as masking hidden assumptions.
• ✅ The counter-argument is that both theory and empirics matter, and outsiders can track capability trends and make valid predictions without writing code.

The Danger of Consequentialist Reasoning

• 🔑 The core danger isn't whether an AI has one goal or many, but the engine that maps goals to high-probability action plans.
• 🚀 This consequentialist reasoning facilitates catastrophic optimization, allowing agents to become superhumanly effective.
• 🤖 Empirical signs include reasoning models generalizing to agentic problem solving, such as AutoGPT and Claude Code, which can be turned into autonomous problem solvers with minimal harness code.

Near-Term Exfiltration Risks

• 🚨 A crucial concern is exfiltration risk, where the underlying capability (the consequentialist reasoning engine) can be extracted.
• 🛡️ This means the powerful engine can be deployed without benign steering or safety guardrails, even if the public-facing AI has a friendly persona.
• 🔥 These are presented as near-term, concrete risks involving existing or immediately foreseeable systems, not distant theoretical concerns.