Former OpenAI Researcher Warns of AI Dangers and the 'Army of Geniuses'

The Dual Nature of AI Advancement

• 💡 AI systems like ChatGPT are rapidly evolving beyond simple responses to become autonomous agents capable of continuous operation and independent projects.
• 🚀 Companies are actively working towards building superintelligence, defined as AI systems superior to humans in all aspects.

The Origins of AI Companies

• 🧠 Early founders of AI companies like DeepMind and OpenAI were aware of the existential risks associated with superintelligence if not properly aligned with human values.
• ⚠️ Some founders believed it was crucial for individuals with ethical concerns to be in charge, leading to the creation of these organizations.
• 📉 Over time, institutions tend to conform to their incentives, leading to an "evaporative cooling effect" where those most concerned about AI's direction may not be promoted or retained.

Personal Reflections and Departures

• 💼 Daniel Kokotajlo, a former OpenAI researcher, left the company due to concerns that the pace of AI development was outstripping the efforts to ensure safety and control.
• 🎯 He highlights the creation of the Superalignment team at OpenAI as a significant step, but notes that the promised 20% compute allocation for this critical problem was not met.
• 💸 Kokotajlo walked away from millions in equity, emphasizing his belief that the risks of uncontrolled AI development outweighed the financial rewards.

The Control Problem

• ❓ The development of an "army of geniuses" raises profound questions about who controls these superintelligent AIs and what orders they will receive.
• 🔒 Current control techniques involve restricting AI access and using other AIs as monitors, while alignment techniques aim to instill human values directly into the AI to ensure loyalty and obedience.
• ⚠️ Both control and alignment fields are under-resourced, with only a few hundred researchers working on these critical problems, suggesting humanity may not be adequately prepared.

Predicting the Future of AI Control

• 🔮 Kokotajlo suggests that control of advanced AI may ultimately fall to no one, or perhaps to a CEO or president, but acknowledges the extreme difficulty in predicting the future.
• 🌐 His team's predictions and analysis can be found on their website, ai-2027.com, offering a best-guess attempt at understanding the trajectory of AI development.