Experiments in Scaling Reinforcement Learning with Verifiable Rewards

Evolution of Reinforcement Learning in LMs

• 💡 The history of RL in LMs includes early successes like TD-Gammon in the 1990s, using neural networks and self-play.
• 🔑 Reward modeling was proposed by DeepMind as a research direction, laying groundwork for RHF system diagrams.
• 🚀 RLHF (Reinforcement Learning from Human Feedback) was crucial for models like InstructGPT and ChatGPT, acting as a catalyst for renewed interest in RL with language models.

Understanding Post-Training Objectives

• 🎯 Supervised Fine-Tuning (SFT) forms the foundation, teaching models formatting and desired response shapes using auto-regressive loss.
• 🧠 Preference Tuning, including algorithms like DPO, uses comparison data to refine model style and abilities like math and code.
• ✨ Reinforcement Fine-Tuning (RLVR) is the most open-ended, allowing models to search for and upweight behaviors useful for solving specific tasks.

Reinforcement Learning with Verifiable Rewards (RLVR)

• ✅ RLVR simplifies the RL agent-environment model, where the language model is the agent, the prompt is the state, and the completion is the action.
• 🛠️ Rewards are verified using scoring functions that check for correctness, such as for math problems, code snippets, or specific text formatting.
• 📊 Reward shaping becomes important for multi-domain problems, allowing for weighting of different intermediate solutions and answers.

Algorithmic Insights and Practicalities

• 🔬 GRPO (Group Relative Policy Optimization) is a key algorithm, closely related to PPO, used in models like DeepSeek R1.
• 📈 Research explores variations like DAPO and Dr. GRPO, focusing on optimizing loss normalization and addressing length bias in token-level losses.
• ⚠️ Implementation and data quality are often more critical for stable and significant performance gains than subtle algorithmic differences, which can introduce variability.

Future Trajectories and Challenges

• 🌱 Expanding RLVR to new domains like tool use, factual knowledge, and agentic behaviors is an active area of research.
• ⚡ Inference time scaling is a significant focus, with RL being a popular method to improve performance by encouraging longer, more detailed responses.
• 🔭 There's potential for post-training to become a central component of language model development, integrating diverse verifiers and RL environments to scale compute and reasoning capabilities.