Running Gemma 3 Models Locally on Mac & Windows with Ollama

Ollama: Easy Local AI Model Deployment

• 💡 Ollama is presented as the simplest method for running AI models locally on your machine.
• 🚀 The project is open-source, allowing users to download binaries or build from source on GitHub.
• 📈 Ollama has seen significant adoption, with tens of millions of developers using it and over 150 million models downloaded.

Under the Hood: Ollama's Inference Engine

• ⚙️ At its core, Ollama functions as a portable inference engine that manages model execution.
• 💻 It handles scheduling models across various hardware like CPUs, GPUs, and TPUs, optimizing memory usage.
• 🛠️ Significant effort has been made to ensure Ollama runs smoothly on Mac and Windows environments, as well as single-GPU Linux systems.

Gemma 3 Integration and Demo

• ✨ Ollama has collaborated with DeepMind to bring Gemma 3 models to its platform, available in four parameter sizes.
• 💬 A demonstration shows running the Gemma 3 4B model on a MacBook Pro, showcasing fast inference without needing API keys.
• 🖼️ Gemma 3's multimodal capabilities are highlighted, successfully interpreting images and even deciphering difficult handwriting from a doctor's note.
• 💨 The 1B parameter model offers even faster inference and requires minimal VRAM (2GB), making it accessible on most machines.
• 📈 The 27B model, while requiring more VRAM, is still usable on medium to high-end MacBooks and other GPU-enabled devices.

Beyond Local: Ollama on Google Cloud

• ☁️ Ollama extends beyond local installations, partnering with Google Cloud Run for serverless deployments.
• ⚡ This allows users to access GPUs on Google Cloud Run, scaling from zero to hundreds of GPUs with rapid startup times (under five seconds).
• ✅ Ollama is available across Windows, Mac, and Linux, enabling powerful AI access both locally and in the cloud.