RewardAnything: Aligning LLMs with Generalizable Principle-Following Reward Models

Challenges of Traditional Reward Models

• ⚠️ Traditional Reward Models (RMs) are rigid, trained on fixed preference datasets, making them struggle to generalize to new tasks or preferences.
• 💸 Adapting to new needs typically requires costly retraining and extensive data collection, limiting flexibility and introducing biases.
• 🔍 These RMs can learn spurious correlations, such as favoring response length over accuracy, and operate as opaque "black boxes" without explicit reasoning.

Introducing RewardAnything: Principle-Following RMs

• 💡 RewardAnything proposes a paradigm shift: RMs that understand and adhere to dynamically provided natural language principles or specifications.
• 🚀 This approach aims for a single, flexible RM that can evaluate diverse contexts simply by changing the principle, similar to instruction-following in LLMs.
• ✅ RewardAnything is a generative RM that produces structured outputs, including reasoning, scores, and a final ranking, enhancing transparency.

RABench: Evaluating Generalization

• 🎯 To measure generalization to unseen principles, the researchers developed RABench, a comprehensive benchmark.
• 📊 RABench uses 50 distinct principles (not used in training) and generates responses from powerful LLMs instructed to follow these principles.
• 🧠 Ground truth is established by multiple LLM judges (e.g., GPT-4.1, Claude 3.7) using a clever consensus mechanism and verified by human annotators.

Advanced Training for Principle Adherence

• 🛠️ RewardAnything is trained using Group Relative Policy Optimization (GRPO), specifically adapted as Group Relative Preference Learning (GRPL).
• 📈 This method teaches the RM to generate accurate evaluations and learn relative preferences through a detailed reward signal, including format and accuracy components.
• 🔬 Ablation studies confirmed the critical importance of explicit principle guidance, listwise preferences, GRPO training, and the RM's explicit reasoning process for performance.

Practical Impact and Future of LLM Alignment

• 🚀 RewardAnything achieved State-of-the-Art performance on traditional benchmarks by explicitly specifying principles (e.g., prioritizing accuracy over length).
• 🤝 A case study demonstrated LLM alignment for nuanced safety behavior using only natural language principles and diverse prompts, without collecting new preference data.
• 🌱 This innovation drastically lowers the barrier to custom AI alignment, enabling flexible and nuanced steering of LLMs with plain language, making AI customization more accessible.