AI Infrastructure: TPUs, Inference, and the Future of Startups with Amin Vahdat

The Scale of AI Compute

• 💡 When performing a deep research query, Google uses thousands of standard servers in collaboration with custom accelerators like TPUs (Tensor Processing Units).
• 🚀 A single TPU chip can pack the general-purpose computing power of 100 servers, with tens of thousands of server equivalents coordinating for complex queries.
• ⚡ These queries often involve multiple sub-queries that are iterated and composed in real-time to provide comprehensive answers.

The Age of Inference

• 🎯 We are currently in the age of inference, with 2025 predicted to be the "Year of Inference," shifting focus from training models to serving them efficiently.
• 📈 Efficiency improvements in AI infrastructure are happening at an exponential pace, with capabilities doubling or increasing tenfold in short periods.
• 💰 Google is proud to be at the frontier of driving down the unit cost of intelligence, passing savings onto customers with significant reductions in cost per query.

Infrastructure and Startup Growth

• 🚀 The historical bottleneck for startups has shifted from funding and hardware to talent and developers.
• 🧠 Cloud computing provided near-infinite capacity, and now AI is multiplying the productivity and capability of existing developers, rather than replacing them.
• 🛠️ Pricing for AI compute and tokens is plummeting, with improvements potentially reaching factors of three or more in a year, unlike the slower pace of storage cost reductions.

The Evolution of TPUs and Transformers

• 💡 Google invented TPUs in 2013 to handle large-scale matrix multiplications for use cases like voice recognition, which required building two more Googles to support 30 seconds of daily voice interaction.
• 🧠 TPUs enabled breakthroughs like transformers by providing immense computing power, with seven generations released, each significantly more capable than the last.
• 📈 The current generation of TPU pods is 10x more capable than the previous one, showcasing rapid advancements in hardware efficiency.

The Rise of AI Agents

• 💬 The most exciting development is the progression of AI agents that can invoke code, interact with other agents, and take action on behalf of users.
• 🧩 These agents are moving beyond generating answers to actively performing tasks, with creativity in their development being highly encouraging.
• 🎯 For venture capital, a supervised associate agent is being developed to help process applications, identify blind spots, and build comprehensive dossiers on startups, enabling more efficient evaluation.