PaliGemma: Adding Vision to Gemma 2 for Multimodal AI

PaliGemma: Extending Gemma with Vision Capabilities

• 💡 PaliGemma is an extension of Gemma 1 and Gemma 2, enabling them to process and understand images.
• 🎯 This multimodal capability allows Gemma to perform tasks like extracting information from images, reading text within images (OCR), and detailed image descriptions.

Advanced Vision Tasks with PaliGemma

• 🔍 PaliGemma introduces zero-shot object detection, allowing it to identify and locate specific objects within an image without prior task-specific training.
• 🧩 It also supports zero-shot segmentation, enabling the model to generate precise masks for specified objects.

Implementation and Model Architecture

• 🧠 The architecture integrates a SigLIP vision encoder with the Gemma LLM, converting images into tokens that serve as input for the language model.
• 📏 PaliGemma offers three model sizes, each with corresponding input resolutions, allowing for a 155x compute range by scaling both model size and image resolution.
• 🛠️ Implementation involves extending Gemma's vocabulary with special tokens for bounding boxes and segmentation masks, and using a small model to upscale segmentation outputs.

Training and Fine-Tuning

• 📚 The core training involves co-training the vision encoder and Gemma LLM on image-text data augmented with OCR, detection, and segmentation pseudo-annotations.
• 🚀 Developers are encouraged to fine-tune PaliGemma on their specific data for optimal performance, with guidance provided in the tech report for hyperparameter selection.
• ✅ While pre-trained models are available, fine-tuning is crucial for achieving great performance on custom tasks.

Applications and Comparisons

• 📈 Performance on academic benchmarks shows varying benefits from larger models versus higher resolutions depending on the task, such as chart QA or outside knowledge VQA.
• 🏆 Fine-tuning PaliGemma has led to state-of-the-art results in areas like table structure recognition, report generation, and molecular structure recognition.
• 📄 Examples like PaliRetrieval demonstrate its use in document retrieval by embedding PDF screenshots, simplifying previous complex pipelines.
• 🤖 PiZero showcases PaliGemma's application in robotics, generating action tokens to control robot actuators for tasks like grinding coffee or folding laundry.

PaliGemma 2 vs. Gemma 3

• ⚖️ PaliGemma 2 offers fixed resolutions and excels at detection and segmentation, while Gemma 3 starts with large resolutions, includes pan and scan for higher resolution, and is a best-in-class text-to-text model retaining multimodal capabilities.
• 💬 Gemma 3 also features an instruction-tuned model for out-of-the-box use and supports interleaved text and data for complex use cases.