Understanding AI Kernels: How AWS Chips Run Code with Emily Webber

What is an AI Kernel?

• 💡 A kernel is fundamentally a user-defined function, which differs from how we typically think about programming in Python.
• 🧠 As you move down the compute stack, especially for hardware like Trainium and Inferentia, kernels become crucial for defining operations directly on the chip.

The Compilation Process

• ⚙️ When programming in PyTorch, the model is compiled through PyTorch XLA (Accelerated Linear Algebra), which represents the model as a graph called an HLO (High-Level Operations) graph.
• 🔗 This HLO graph is then fed into a compiler that converts it into an instruction set executable directly on the hardware.
• 🛠️ A kernel allows users to override the compiler and define these chip operations themselves using a specific kernel library.

Kernel Library and Usage

• 🚀 AWS's kernel library is called Nikki (Neuron Kernel Interface).
• 🎯 Kernels can range from simple functions like tensor addition to defining entire model passes (forward or backward).
• 📈 Developing kernels involves studying compute optimization, including data movement, memory utilization, and compute utilization.
• 💻 The goal is to implement algorithms as kernels to improve performance, particularly for workloads like large language models.