Stella Li: Spurious Rewards - Rethinking Training Signals in RLVR

Surprising RLVR Findings

• 💡 Reinforcement Learning with Verifiable Rewards (RLVR) shows significant gains, especially with Qwen-Math models, even when trained with uninformative or incorrect reward signals.
• 🎯 Experiments used "spurious rewards" like majority voting, format-only, incorrect labels, and random rewards, all yielding performance improvements.
• 🔑 This suggests that RL might primarily elicit existing behaviors rather than teaching new fundamental reasoning abilities, particularly at academic scales.

Model-Specific Behaviors

• 🧠 Qwen-Math models exhibit a strong tendency for "code reasoning", generating precise Python code to solve math problems, which is highly correlated with accuracy.
• 🚀 RL training, even with spurious rewards, increases the frequency of code reasoning in Qwen-Math models, acting as a performance shortcut.
• ⚠️ In contrast, "bad code models" (e.g., Qwen-Base, OMO SFT) that are less proficient at code reasoning, show a decrease in code frequency when trained with meaningful rewards, shifting towards language reasoning for better accuracy.

The Role of Random Rewards and Prompts

• 🔬 The clipping term in the GRPO objective is crucial for random rewards to work in Qwen-Math, as it amplifies high-probability tokens (like code reasoning tokens) during training.
• ✅ Disabling this clipping term or manipulating batch sizes to prevent clipping eliminates performance gains from random rewards.
• 💬 Simple prompts instructing Python usage or rewarding for the string "Python" also lead to significant performance improvements in Qwen-Math, further demonstrating the elicitation of pre-existing capabilities.

Implications for RL Research

• 📈 Researchers should validate proposed RLVR algorithms across multiple model families and against spurious baselines to accurately assess true reasoning gains.
• 🌱 Much of the observed gains from academic-scale RL may stem from eliciting pre-existing capabilities, suggesting a focus on instilling desirable behaviors during pre-training for more effective post-training.
• 📊 Transparency in training data and pipelines is crucial for understanding diverse model behaviors and advancing scientific insights in RL.