Scaling AI Rewards: How DeepSeek-GRM Redefines Generalist Reward Modeling

Reward modeling (RM) is the backbone of reinforcement learning (RL) for large language models (LLMs), guiding them to align with human values, reason effectively, and adapt to diverse tasks. But crafting accurate reward signals for general queries—where no clear rules or ground truth exist—has been a persistent challenge. A new preprint from DeepSeek-AI, titled Inference-Time Scaling for Generalist Reward Modeling, introduces a groundbreaking approach called Self-Principled Critique Tuning (SPCT) and the resulting DeepSeek-GRM models. This work explores how to make reward models more flexible and scalable at inference time, offering insights that could reshape how we optimize LLMs. Here’s a summary of the key takeaways and why they’re exciting.

The Challenge of Generalist Reward Modeling

Traditional reward models excel in specific domains like math or coding, where verifiable rules or human-designed environments provide clear feedback. However, general domains—think open-ended questions or creative tasks—require rewards that capture diverse, nuanced criteria without explicit references. The paper identifies four key hurdles for generalist RM:

1. Input Flexibility: Handling single, paired, or multiple responses seamlessly.
2. Reward Quality: Generating accurate rewards across varied domains.
3. Inference-Time Scalability: Improving reward quality with more compute during inference.
4. Scalable Behaviors: Learning to leverage extra compute effectively.

Existing methods, like scalar (single-number scores) or pairwise (comparing two responses) reward models, often fall short. Scalar models lack diversity in outputs, limiting scalability, while pairwise models struggle with single-response inputs. The authors propose a new path: pointwise generative reward modeling (GRM) combined with a novel training method.

Enter SPCT and DeepSeek-GRM

The core innovation is Self-Principled Critique Tuning (SPCT), a two-phase training strategy that empowers GRMs to generate high-quality rewards. Here’s how it works:

Phase 1: Rejective Fine-Tuning

This “cold start” phase trains the model to produce principles (guiding criteria) and critiques (textual judgments) in a consistent format for any number of responses. Using a pointwise GRM, the model samples multiple trajectories for each query-response pair, rejecting those that misalign with ground truth or are too easy. This ensures flexibility and sets the stage for more advanced learning.

Phase 2: Rule-Based Online RL

SPCT then employs reinforcement learning with a rule-based reward system (GRPO) to optimize both principle and critique generation. Unlike prior methods, it avoids format-specific rewards, using a high KL penalty to maintain output consistency. The model learns to generate adaptive principles—tailored to the query and responses—and accurate critiques, boosting reward quality in general domains.

The result is DeepSeek-GRM-27B, built on Gemma-2-27B, which generates rewards as textual critiques with extractable scores (1–10). This generative approach allows richer, more flexible reward signals compared to scalar alternatives.

Scaling at Inference Time

To make rewards more accurate, DeepSeek-GRM leverages inference-time scaling through parallel sampling:

• Voting: The model generates multiple sets of principles and critiques (up to 32 samples), summing their scores to produce finer-grained rewards. More samples expand the reward space, capturing diverse judgment perspectives.
• Meta RM: A secondary model filters low-quality samples by scoring the correctness of principles and critiques, ensuring only the best contribute to the final vote. This guided voting achieves top performance with just 8 samples.

This approach mimics how humans refine judgments by considering multiple angles, making the model’s rewards more robust and precise.

Key Results

The paper’s experiments are compelling:

• Benchmark Performance: DeepSeek-GRM-27B outperforms baselines like LLM-as-a-Judge and DeepSeek-PairRM across Reward Bench, PPE, RMB, and ReaLMistake benchmarks, achieving an overall score of 69.9 (greedy decoding) and 72.8 with meta RM-guided voting at 32 samples. It rivals larger models like GPT-4o (71.3).
• Inference vs. Training Scaling: With 32 samples, DeepSeek-GRM-27B matches or exceeds the performance of a 671B MoE model (greedy decoding), showing that inference-time scaling can outpace training-time model size increases (see Figure 4 in the paper).
• Principle Power: A preliminary test showed that high-quality principles boost reward accuracy significantly (Table 1), validating SPCT’s focus on principle generation.
• No Severe Biases: The model maintains balanced performance across domains, reducing the risk of skewed judgments.

Why This Matters

For Researchers

SPCT and DeepSeek-GRM highlight the potential of generative reward models to unify input flexibility and scalability. The finding that inference-time scaling can rival training-time scaling challenges the “bigger is better” paradigm, suggesting compute-efficient alternatives for RM. The meta RM’s role in filtering noisy samples also opens avenues for hybrid RM architectures.

For Developers

The flexibility of pointwise GRMs makes them a versatile tool for post-training LLMs or guiding inference-time search. Open-sourcing DeepSeek-GRM invites developers to experiment with generalist RM in real-world applications, from chatbots to creative writing aids.

For AI Safety

The paper’s ethics statement acknowledges risks like bias amplification if training data is flawed. SPCT’s principle generation, while powerful, needs careful oversight to avoid unintended biases. The call for human-in-the-loop frameworks and transparent evaluation is a step toward responsible deployment.

Limitations and Future Directions

DeepSeek-GRM isn’t perfect. It faces efficiency challenges with large-scale sampling, and some tasks (like reasoning-heavy ones) remain tough, as seen in DeepSeek-R1’s underwhelming performance. The authors suggest future work could:

• Integrate GRMs into online RL pipelines as reward system interfaces.
• Explore co-scaling with policy models for synergistic improvements.
• Enhance efficiency for practical deployment.

Final Thoughts

DeepSeek-GRM and SPCT mark a leap toward generalist reward modeling that’s both flexible and scalable. By generating adaptive principles and critiques, the model mimics human-like judgment, offering a smarter way to guide LLMs. The fact that a 27B model can compete with a 671B behemoth through inference-time tricks is a testament to the power of clever design over brute force.

For anyone working on AI alignment or LLM optimization, this paper is a must-read. It’s a reminder that sometimes, the best rewards come not from bigger models, but from smarter ways to scale what we’ve got.

What do you think—could inference-time scaling redefine how we train AI, or is model size still king? Drop your thoughts below!