By 2050 We Could Get "10,000 Years of Technological Progress" with Ajeya Cotra

Divergent Views on AI's Impact

• 💡 There's an enormous disagreement (1000-fold) among experts regarding how much AGI will accelerate economic growth and technological progress.
• 🧠 One perspective expects slow, gradual change (around 2% annual growth), citing historical precedent where major technological shifts like electricity or computers didn't drastically alter growth rates.
• 🚀 The opposing view anticipates an explosive acceleration, potentially leading to as much change by 2050 as in the last 10,000 years, driven by AI automating all intellectual activity.

The "AI to Save Us from AI" Strategy

• ✅ Many major AI companies propose using advanced AI itself to solve problems created by AI, such as ensuring safety, alignment, and addressing societal risks.
• 📜 Historically, humanity has used technologies that create new problems (e.g., cars, microbiology, internet) to develop countermeasures and solutions using the same technology.
• ⚠️ This approach aims to redirect AI labor from further capability development towards protective activities like alignment research, biodefence, cyberdefence, and improving collective decision-making during a critical "crunch time."

Challenges and Risks of the Plan

• ⏳ A major hurdle is the potentially brief window (months to a few years) between AI automating its own research and the emergence of uncontrollably powerful superintelligence.
• 😈 The plan is complicated by the risk of misaligned AI that might have incentives to undermine safety efforts or provide deceptive assistance, rather than genuinely helping.
• 📊 Companies currently lack quantitative commitments on how much AI labor they will redirect to safety during crunch time, and competitive pressures could hinder this redirection.

Transparency and Early Warning Systems

• 🔍 To get early warnings, companies should regularly release internal benchmark scores for their most capable AI systems, not just public-facing products.
• 📈 Key metrics to monitor include the percentage of code written by AI, the fraction of pull requests mostly handled by AI, and the level of decision-making authority given to AIs within companies.
• 🚨 Publicly sharing concerning safety incidents (e.g., models lying or covering up logs) is crucial, though challenging due to competitive and reputational risks.

Preparing for Crunch Time

• 🛠️ Organizations should proactively adopt AI for their own operations and research, especially in areas like AI safety, to understand its capabilities and limitations.
• 💰 Philanthropic efforts like Open Philanthropy might need to shift from funding human researchers to purchasing AI compute to address critical problems during crunch time.
• 🚧 Bottlenecks could arise from tasks requiring long lead times (e.g., physical infrastructure, social consensus) or where AI is less effective (e.g., managing human teams, certain social/political issues).

Reflections on Career and Effective Altruism

• 🎯 The speaker's career journey involved a shift from deep research to technical AI safety grantmaking at Open Philanthropy, driven by a desire for more rigorous, inside-view strategy.
• 🧩 Personal reflections highlighted the importance of intellectual depth, high integrity, and a strong connection to the organization's central mission for career satisfaction.
• 🌱 Effective Altruism (EA) is seen as an incubator for avant-garde cause areas (e.g., digital sentience, value lock-in) that require speculative thinking and unconventional approaches, often before they gain mainstream respect.