Running Andrej Karpathy’s LLM Council GitHub Repo (AI Quietly Picked MS Dhoni)

Understanding the LLM Council Concept

• 💡 The LLM Council, inspired by Andrej Karpathy, involves multiple AIs debating a question, critiquing each other's responses, and a chairman AI synthesizing a final verdict.
• 🎯 The core idea is to achieve more nuanced and balanced answers through collaboration, moving beyond single-AI computation.

How the Council Operates

• ⚙️ The process unfolds in three stages: a query sent to all council members, a review where AIs anonymously score each other's answers (lower score is better), and a verdict where a chairman AI synthesizes the final response.
• 🚀 Setting up the system is surprisingly straightforward using a GitHub repository, requiring code download, dependency installation, OpenRouter API key configuration, and running a startup script.
• 🤖 The default council includes powerful models like GPT-4o, Gemini 1.5 Pro (as chairman), and Claude 3 Opus, with the flexibility to customize the lineup.

Testing with a Complex Question

• 🏏 The council was tested with a subjective and complex question: "Who is the best captain of the men's Indian cricket team?", a topic known for sparking debate.
• 📊 During the peer evaluation, GPT-4o received the lowest average score (1.8), indicating its AI peers collectively judged its initial answer as the most superior.

Emergent Consensus and Nuance

• 🧠 The chairman AI, Gemini 1.5 Pro, delivered a multi-faceted answer, identifying captains by their unique strengths rather than just picking one name.
• ✅ The final verdict, after considering various angles, concluded that the overall consensus leaned towards MS Dhoni, primarily due to his success in winning World Cups.
• ✨ This architecture transforms a simple query into a structured debate, yielding a more balanced, detailed, and nuanced conclusion than a single model could produce.