Google AI Edge: Small Language Models on Mobile and Web

Bringing Language Models to the Edge

• 💡 Small language models are being brought to mobile and web devices, offering benefits like real-time low latency processing, offline use, and enhanced privacy advantages.
• 🚀 Google AI Edge provides tools for developers to select the right solution, from high-level APIs to direct runtime interfaces for custom AI features.

Model Customization and Optimization

• 🛠️ Developers can leverage the LiteRT community on Hugging Face to find and contribute models ready for edge deployment, including recent DeepSeek and Jamma models.
• 🧠 Customization involves using edge-optimized Python implementations, fine-tuning with tools like TorchTune or Hugging Face Transformers, and then converting to an edge model format.
• 🗜️ Quantization using AI Edge Quantizer reduces model size and improves latency, with new support for INT4 post-training quantization offering up to an eight-time size reduction with minimal quality loss.
• 🏗️ The AI Edge Torch Generative API allows for architectural changes or bringing new language models to the edge by combining PyTorch and AI Edge Torch layers.

Inspecting and Testing Edge Models

• 🔍 Model Explorer is a visualization tool for understanding model structure, allowing side-by-side comparisons of different model variants or quantization schemes.
• ☁️ AI Edge Portal is a GCP service for testing models on a variety of real devices in a cloud lab, enabling benchmarks and bulk inference to mirror production environments.

Advanced Edge AI Capabilities

• 📚 On-device Retrieval Augmented Generation (RAG) allows small language models to access and utilize specific knowledge during inference, parsing, storing, and retrieving data from user storage.
• 🎯 The Google AI Edge RAG SDK includes an on-device embedding model, chunking, vector database, and retrieval function, available on Android with iOS and web support coming.
• ⚡ Function calling (tool use) enables language models to take actions by executing registered application functions or APIs, with a new SDK that supports customization for improved accuracy.

Infrastructure and Performance Improvements

• 🚀 LiteRT simplifies developer experience, with a GPU-first runtime stack designed for significant performance increases.
• 📈 The ML drift GPU delegate offers up to a 12x improvement for small language models on device compared to other inference frameworks.
• 📱 NPUs are increasingly important for AI on mobile, offering significant performance gains (e.g., 25x over CPUs, 8x over GPUs) and reduced resource utilization for a more responsive and power-efficient user experience.
• 🤝 Partnerships with Qualcomm and MediaTek, unified by LiteRT, streamline NPU integration and binary compilation for various vendors.

Real-World Applications and Partner Success

• 🔒 NSING Infocurity reduced deep fake detection time from 67 seconds to 1.2 seconds using LiteRT quantization and delegates, improving efficiency and accessibility.
• 🗣️ Argmax achieved a 30% performance improvement by switching to LiteRT and its ML drift GPU delegate for their on-device inference SDKs.
• 🎶 Audio Shake leverages LiteRT's performance for real-time audio separation use cases, enabling advanced features with a small team.