GDPO: Decoupling Rewards for Stable Multi-Reward RL in LLMs

The Challenge of Multi-Reward RL

• 🧠 Large Language Models (LLMs) need to follow multiple, often conflicting, rules simultaneously, such as accuracy, politeness, specific formats, and length constraints.
• ⚠️ Standard reinforcement learning (RL) methods struggle when models are presented with diverse and sometimes contradictory goals.

GRPO's Reward Signal Collapse

• 📉 The previous method, Group Relative Policy Optimization (GRPO), combines all reward scores before calculating the 'advantage value' (the teaching signal).
• 💡 This process leads to reward signal collapse, where distinct outcomes appear identical to the AI, resulting in a weak training signal.
• ❌ Consequences include suboptimal convergence (slow, inaccurate learning) and even early training failure, where models stop improving.

GDPO's Decoupled Normalization

• ✅ Group reward-Decoupled Normalization Policy Optimization (GDPO) resolves this by calculating the advantage value for each individual reward separately.
• 🚀 Only after individual normalization are these separate advantage values summed, preserving the resolution and providing a clear, precise teaching signal.
• ✨ This approach gives the model a crystal clear message, allowing it to understand nuanced differences between good and great attempts.

GDPO's Performance Advantages

• 📊 In tool calling tasks, GDPO significantly improved overall accuracy and achieved over 4% better format compliance compared to GRPO.
• 📈 For math reasoning problems with conflicting rewards (accuracy vs. length), GRPO showed training collapse, while GDPO maintained stability, continuously improving accuracy and reducing