Tri Dao on AI Inference Costs, Nvidia's Dominance, and Future Speed

AI Inference Evolution & Cost Reduction

• 💡 Since ChatGPT's launch, inference costs have dropped 100x, driven by advancements in modeling and optimization techniques.
• 🚀 Key optimizations include quantization (e.g., 4-bit models in GBD OSS) to reduce memory footprint and hardware-software co-design (like Flash Attention) to minimize data movement.
• 🧠 Flash Attention specifically addressed memory access bottlenecks by rewriting the attention algorithm for efficiency.

Shifting AI Hardware Landscape

• 🎯 Nvidia currently dominates with 90% of AI workloads, but this is predicted to diversify to multi-silicon (AMD, Google, Amazon, Cerebras, Grock) within 2-3 years.
• 🛠️ Chip design faces challenges due to evolving architectures (e.g., Mixture of Experts for sparsity) and diversifying workloads beyond traditional chatbots.
• 📈 Future hardware will specialize for distinct needs: low-latency agentic systems, high-throughput batch processing, and interactive chatbots.

AI's Impact on Optimization

• 🤖 AI models, particularly Claude Code, have significantly boosted productivity (1.5x) for GPU kernel optimization by assisting humans.
• ⚠️ Fully automatic GPU kernel generation is still early due to a lack of expert-level training data, which is difficult to acquire or synthesize.
• 🔍 Future models need enhanced agentic capabilities to use tools effectively and identify when to seek new information.

Next-Gen Architectures & Workloads

• ⚡ Another 10x reduction in inference cost is anticipated from hardware specialization, advanced model architectures (e.g., Mamba), and improved kernel optimization.
• 🧩 Mixture of Experts (MoE) and State Space Models (Mamba) are crucial architectural innovations for achieving AGI at reasonable costs, especially for large batch inference by compressing history (KV cache).
• 📊 Evolving workloads include agentic systems (AI taking actions, using tools) and real-time video generation, which demand different optimization profiles and significant compute.

Future AI & Research Directions

• 🌱 The goal is to achieve expert-level AI for economically valuable tasks, moving beyond median human performance.
• 🤖 Robotics is a key research area, requiring solutions for data problems in actuation and multi-resolution/multi-time scale processing for control and planning.
• ✅ The speaker values both academic exploration (long-term, speculative research) and industry execution (fast-paced, market-driven product development) for advancing AI.