Gemma on Mobile & Web: Best Practices for On-Device AI with MediaPipe

Why On-Device AI?

• 💡 Running AI models on mobile and web devices offers significant advantages over cloud-based solutions, primarily due to privacy, latency, cost, and offline availability.
• 💰 Eliminating cloud bills enables different business models, especially for free or single-payment apps, by avoiding variable costs associated with user LLM interactions.
• 🔒 On-device processing ensures that sensitive user data remains on the device, enhancing privacy and security.
• ✈️ Offline availability means applications can function reliably even without an internet connection, crucial for travel or areas with poor signal.
• ⚡ Reduced latency provides a faster response time for users, as data doesn't need to travel to a server and back, especially beneficial with smaller, optimized models.

System vs. In-App AI Models

• 🤖 System AI (e.g., Gemini Nano on Android, Apple Intelligence on iOS) integrates LLMs directly into the platform, offering large, capable models that are easy to use but limited in customization.
• 📱 In-app AI involves bundling models directly into your application, providing full flexibility, customization, and fine-tuning capabilities.
• ⚖️ While system models offer convenience and power, in-app models provide greater control and can be optimized for specific use cases and a wider range of devices.

Gemma 3 1B: A Powerful In-App Model

• ✨ Gemma 3 1B is highlighted as an exceptional in-app model, particularly for its size and performance.
• 📦 Quantized, it's only 529 megabytes, making it easy to download over the air, and under 400 megabytes when zipped.
• 💻 It can run on devices with as little as 4GB of RAM, making it accessible on a vast majority of user devices.
• 🚀 Performance is impressive, with speeds of 2500 tokens per second for pre-fill and 56 tokens per second for decoding, feeling almost instant for many tasks.

Use Cases and Best Practices

• 🗣️ Natural Language UI Control: Applications can be driven by natural language commands, allowing users to interact with features like flight booking or chart customization without navigating complex menus.
• ✍️ Text Generation: Powerful applications can be built for tasks like data captioning (e.g., workout summaries), summarization of in-app content, and generating smart replies for messaging.
• 🎮 In-game Dialogue: NPCs can generate custom, context-aware dialogue, enhancing player immersion.
• 🎯 Customization is Key: For smaller models like Gemma 1B, few-shot prompting or fine-tuning on custom data significantly improves results.
• 🌐 Retrieval Augmented Generation (RAG): For querying large datasets, RAG on-device, supported by the new Google AI Edge RAG SDK, allows for efficient retrieval and coherent response generation.

Integrating Gemma with MediaPipe

• 🛠️ The MediaPipe LLM Inference API provides a cross-platform solution for running Gemma on Android, iOS, and Web.
• 🚀 It offers high performance by optimizing for both CPU and GPU, a rare feature on mobile platforms.
• 💻 Integration is simplified with just a few lines of code in Kotlin, Swift, or JavaScript, making it accessible for developers to configure, initialize, and run Gemma within their applications.