Gemma 3 Architecture and Design: Tokenizers, Model Sizes, and Long Context

Tokenizer Improvements for Multilingualism

• 💡 The tokenizer is the crucial input/output layer of a language model, translating numbers into words.
• ⚠️ Previous tokenizers struggled with non-Latin languages, splitting words into multiple tokens.
• 🚀 Gemma 3 features a significantly larger tokenizer (256,000 tokens) with improved coverage for 64 languages, enhancing multilingual ability and effective sequence length.

Gemma 3 Model Sizes and Quantization

• 📊 Gemma 3 offers four sizes: 1B, 2B, 9B, and 27B, with increased capacity in the 2B and 9B models for multimodal and multilingual capabilities.
• ⚙️ Model sizes were designed with quantization in mind, optimizing performance on popular hardware rather than just float16 setups.
• ⚖️ The 12B model is highlighted as a strong balance between performance and cost, while the 1B model is specialized for tasks like text classification and sentence rewriting.

Predictable Scaling and Long Context

• 📈 A key innovation in Gemma 3 is predictable scaling across models (4B to 27B), ensuring consistent multimodal capabilities, context length, and training regimes.
• 📚 The context length has been extended from 8,000 tokens in Gemma 2 to 128,000 tokens, enabling the processing of approximately 500 pages of text or an entire codebase.
• 🛠️ The long context recipe is designed to be simple and fine-tuning friendly, requiring minimal code changes by tuning only two parameters.

Efficient Long Context Inference

• ⚡ Gemma 3 employs a hybrid attention mechanism with a ratio of five local attention layers for each global attention layer.
• 🧠 This approach significantly reduces the GPU memory requirements for inference compared to similar models, making it more cost-effective.
• 📉 The new architecture allows models to maintain performance down to 128k tokens and beyond, with 12B and 27B models performing well up to 256k and even 500k tokens respectively.