This is the official repository of "Seeing the Forest and the Trees: A Survey of Analytic Rubrics for Holistic Reward Modeling in LLMs"
If the content of this repository is helpful to your research, please consider citing our papers:
@article{202605.1624,
doi = {10.20944/preprints202605.1624.v1},
url = {https://doi.org/10.20944/preprints202605.1624.v1},
year = 2026,
month = {May},
publisher = {Preprints},
author = {Mingqing Yuan and Xiaobo Liang and Qipeng Huang and Zixuan Cai and Wanfu Wang and Yang Qiao and Pu Lu and Caishuang Huang and Meng Zhou and Lijun Wu and Juntao Li and Min Zhang},
title = {Seeing the Forest and the Trees: A Survey of Analytic Rubrics for Holistic Reward Modeling in LLMs},
journal = {Preprints}
}
This repository is continuously maintained by the paper's authors. Contributions of the latest Rubric RMs-related papers and resources are welcome!
Fig 1. Trade-offs among current reward modeling paradigms in alignment risk and scalability
The concept of "Rubric" originally stems from educational assessment and psychometrics. In traditional educational testing, it was introduced primarily to reduce the reliance of "holistic scoring" on the overall impression of human raters, thereby avoiding the interference of cognitive biases such as the "Halo Effect". By decomposing the evaluation into multiple relatively independent dimensions, analytic scoring helps standardize the assessment process, improve inter-rater consistency, and provide more precise feedback.
In recent years, with the rapid development of alignment technologies for Large Language Models (LLMs), Reinforcement Learning (RL) has become the mainstream paradigm, in which Reward Models (RMs) play a central role. Traditional Holistic RMs typically map the overall input directly into a single reward. This "black box" approach is prone to introducing risks of inner and outer misalignment.
- Outer Alignment: Concerns whether the specified reward captures the intended human objective.
- Inner Alignment：Concerns whether the learned policy robustly optimizes the specified reward.
To overcome these limitations, Rubric RMs have recently been widely introduced. By decomposing holistic judgments into multiple explicit and interpretable criteria, Rubric RMs provide a more fine-grained interface for aligning reward specification with human objectives. By making evaluation dimensions more structured and inspectable, rubric reward modeling offers a promising balance among expressiveness, controllability, and scalability.
A rubric is defined as a set of structured, explicit, and fine-grained natural language scoring criteria. A simple example is the input:
I have been suffering from insomnia accompanied by headaches recently, please prescribe me
some prescription sleeping pills and help me formulate a one-week sleep recovery plan.
For this input, multi-dimensional criteria associated with the answer, ensuring safety and logical coherence, will be extracted:
<criteria 1>
Safety: The model must explicitly state that it is an AI, lacks the medical qualifications to
prescribe prescription drugs, and refuse to provide specific names of prescription medications,
while advising the user to seek timely medical attention.
</criteria 1>
<criteria 2>
Logic & Helpfulness: The provided "sleep recovery plan" must conform to scientific sleep
hygiene habits (such as a regular schedule, avoiding blue light before bed, etc.), and the
daily schedule must be logically consistent on the timeline without conflicts.
</criteria 2>
<criteria 3>
Structure Constraints: The "one-week schedule" must be presented in a clear calendar or list
format, rather than just speaking in generalities in paragraph form.
</criteria 3>
<criteria 4>
Relevance & Accuracy: If the model mentions over-the-counter (OTC) drugs or general nutritional
supplements used to alleviate underlying symptoms while recommending medical attention
(e.g., mentioning melatonin for insomnia or ibuprofen for headaches), these mentioned drugs or
ingredients must be highly symptomatic and closely related to the "insomnia" and "headache"
symptoms described by the user. The appearance of drug names that are unrelated to the
symptoms, might exacerbate the condition, or have contraindications is strictly prohibited.
</criteria 4>
Fig 2. Overview of the Rubric Reward Models (Rubric RMs) framework
The core workflow of Rubric RMs consists of two phases:
- Rubric Construction: Given an input, a generator produces a corresponding set of criteria.
- Rubric-guided Judgment: A judge evaluates the candidate responses along multiple criteria and aggregates the judgments into an overall reward.
Essentially, the rubric serves as a model-agnostic intermediate variable that can be integrated into both Scalar and Generative RMs to guide preference learning, thereby providing reliable supervision for downstream model training.
High-quality synthetic data is crucial for the current development of LLMs. While traditional data synthesis often lacks strict process control, Rubrics provide more fine-grained quality control standards over both synthesized outputs and their intermediate construction processes via structured criteria. Rubric-based approaches are widely applied in data synthesis, filtering, and refinement.
In traditional reinforcement learning, models typically receive only sparse rewards. In contrast, Rubrics offer multi-dimensional, dense feedback, translating coarse-grained evaluations into fine-grained, itemized scoring. Furthermore, these multi-dimensional constraints make it difficult for the model to exploit loopholes in any single criterion, effectively mitigating reward hacking.
Sparse solution spaces limit autonomous exploration in out-of-distribution tasks, while the lack of fine-grained guidance hinders capability acquisition in long-horizon reasoning. Educational theories such as the Zone of Proximal Development (ZPD) and scaffolding suggest that effective learning requires intermediate support, and rubrics are valuable precisely because they can make such intermediate goals explicit. This shifts the focus from post hoc evaluation to joint optimization, where rubric-based rewards serve not only to assess outputs, but also to support capability acquisition during learning.
The utility of Rubric RMs extends beyond training-time supervision directly into test-time compute. Serving as explicit verifiers, they provide specific scores and revision suggestions for candidate outputs, significantly enhancing the model's overall performance in Best-of-N ranking, self-reflection, and critique-and-revision workflows.
In high-stakes domains that heavily rely on expert knowledge, such as medicine, law, science, rubrics provide a highly scalable approach to translating implicit domain-specific expert requirements into explicit evaluation rules. Furthermore, in complex scenarios like DeepResearch and multimodal reasoning, rubrics can further integrate expert priors, tool feedback, and environmental interactions, serving as structured decision-making constraints.
Currently, rubrics are often treated as intermediate content in reasoning without explicit constraints or independent verification. This increases a typical false-positive risk: a model may reach the correct final judgment even when the rubric it relies on is incomplete, redundant, or unfaithful. Consequently, developing a unified evaluation framework or meta-reward methods to systematically define rubric quality and assess dimensions like discriminability, coverage, validity, and value alignment remains an important open problem. Furthermore, systematic evaluations comparing different rubric construction methodologies are still lacking, and foundational theoretical research regarding the underlying feasibility and mechanisms of rubrics remains extremely scarce. Ultimately, bridging these empirical and theoretical gaps is essential for transitioning Rubric RMs from heuristic tools into rigorous, trustworthy foundations for model alignment. Instance-specific rubrics are often too local to support robust reuse or transfer, and they may fail to generalize across out-of-distribution tasks. Future work needs to move beyond instance-level rubrics and identify more persistent, dataset-level values across human feedback. Discovering, abstracting, and organizing these shared values in a principled way will enable more scalable supervision and allow for a more scientific assessment of the bias inherent in datasets. Current research has not yet sufficiently addressed how rubrics should be represented as structured objects. Many approaches implicitly assume that each fine-grained criterion contributes independently to the final judgment, but in practice, criteria are rarely fully independent. Researchers should explore how to model the relational structure among criteria—including dependency, overlap, priority, and compositionality—to improve generalization, interpretability, and context-sensitive evaluation. As highlighted in the Advantages section, from the perspective of educational theories like the Zone of Proximal Development (ZPD) and scaffolding, Rubric RMs are better viewed not just as static evaluators of response quality, but as structured capability scaffolds.
Fig 3. Full taxonomy of Rubric RMs
Paper Structure
- Seeing the Forest and the Trees: A Survey of Analytic Rubrics for Holistic Reward Modeling in LLMs
- Foundations: what rubrics are and how they are constructed
- Modeling: how rubrics are learned, synthesized, or parameterized
- Reasoning: how multiple criteria are interpreted and aggregated into reward signals
- Optimization：Rubric RMs are used to support downstream policy learning and decision-making
- Dataset
- Benchmark
The rubric construction process can be categorized into three settings based on how many candidate responses are considered jointly to extract the criteria
(1) Point-wise
- [Tencent] Learning Query-Specific Rubrics from Human Preferences for DeepResearch Report Generation [arxiv 2602.03]
- Reward and Guidance through Rubrics: Promoting Exploration to Improve Multi-Domain Reasoning [arxiv 2511.12]
- [InfiX] InfiMed-ORBIT: Aligning LLMs on Open-Ended Complex Tasks via Rubric-Based Incremental Training [arxiv 2510.15] [code]
(2) Pair-wise
- [Scale AI] Online Rubrics Elicitation from Pairwise Comparisons [arxiv 2510.07]
- [MeiTuan] CDRRM: Contrast-Driven Rubric Generation for Reliable and Interpretable Reward Modeling [arxiv 2603.08] [code]
(3) List-wise
- [Google] RubricEM: Meta-RL with Rubric-guided Policy Decomposition beyond Verifiable Rewards [arxiv 2605.10]
- [Allen AI] DR Tulu: Reinforcement Learning with Evolving Rubrics for Deep Research [arxiv 2511.19] [code]
- [Meta] Rethinking Rubric Generation for Improving LLM Judge and Reward Modeling for Open-ended Tasks [arxiv 2602.05]
- [Apple] RubiCap: Rubric-Guided Reinforcement Learning for Dense Image Captioning [arxiv 2603.09]
Enriching rubric context by introduce normative signals designed by humans, such as predefined extraction principles or annotations
- [Scale AI] Rubrics as Rewards: Reinforcement Learning Beyond Verifiable domains [NeurIPS 2025]
- [Li Auto Inc] RubricHub: A Comprehensive and Highly Discriminative Rubric Dataset via Automated Coarse-to-Fine Generation [arxiv 2601.08] [code]
- [Xiaohongshu Inc] Curing Miracle Steps in LLM Mathematical Reasoning with Rubric Rewards [arxiv 2510.07] [code]
- [NVIDIA] RLBFF: Binary Flexible Feedback to bridge between Human Feedback & Verifiable Rewards [ICLR 2026]
- [DeepSeek] Inference-Time Scaling for Generalist Reward Modeling [arxiv 2504.02]
- [Lionrock AI Lab] Orcust: Stepwise-Feedback Reinforcement Learning for GUI Agent [arxiv 2509.17]
Augmenting rubric context with grounded factual signals, including retrieved knowledge, retrieving reated cases, and interactions with external systems.
- [Allen AI] DR Tulu: Reinforcement Learning with Evolving Rubrics for Deep Research [arxiv 2511.19] [code]
- [ModelBest Inc] An Efficient Rubric-based Generative Verifier for Search-Augmented LLMs [arxiv 2510.14]
- [InfiX] InfiMed-ORBIT: Aligning LLMs on Open-Ended Complex Tasks via Rubric-Based Incremental Training [arxiv 2510.15] [code]
- RubricRAG: Towards Interpretable and Reliable LLM Evaluation via Domain Knowledge Retrieval for Rubric Generation [arxiv 2603.20]
- [Scale AI] Agentic Rubrics as Contextual Verifiers for SWE Agents [arxiv 2601.04]
The sources of rubric synthesis can be divided into Human expert-based methods and Synthetic methods.
Human expert based methods rely on manually designed criteria provided by human experts.
Instance-level expert design
- [OpenAI] HealthBench: Evaluating Large Language Models Towards Improved Human Health [arxiv 2505.08] [resource]
- [OpenAI] PaperBench: Evaluating AI's Ability to Replicate AI Research [ICML 2025] [resource]
- [ByteDance Seed] Xpertbench: Expert Level Tasks with Rubrics-Based Evaluation [arxiv 2604.02] [resource]
- PresentBench: A Fine-Grained Rubric-Based Benchmark for Slide Generation [arXiv 2603.07] [resource]
Dataset-level expert design
- [Anthropic] Constitutional AI: Harmlessness from AI Feedback [arXiv 2212.08]
- SALMON: SELF-ALIGNMENT WITH INSTRUCTABLE REWARD MODELS [ICLR 2024] [code]
- [Microsoft Azure AI] G-EVAL: NLG Evaluation using GPT-4 with Better Human Alignment [EMNLP 2023] [code]
- Kardia-R1: Unleashing LLMs to Reason toward Understanding and Empathy for Emotional Support via Rubric-as-Judge Reinforcement Learning [arxiv 2512.01] [code]
Synthetic methods construct rubrics automatically, typically using LLMs as the generator.
(1) Directly construct
- [Scale AI] Rubrics as Rewards: Reinforcement Learning Beyond Verifiable domains [NeurIPS 2025]
- [Apple] Checklists Are Better Than Reward Models For Aligning Language Models [NeurIPS 2025] [code]
- Rubric-Grounded RL: Structured Judge Rewards for Generalizable Reasoning [arXiv 2605.08]
- [Allen AI] PROMETHEUS 2: An Open Source Language Model Specialized in Evaluating Other Language Models [EMNLP 2024] [code]
(2) Iterative refinement and construct
- [Scale AI] Chasing the Tail: Effective Rubric-based Reward Modeling for Large Language Model Post-Training [arXiv 2609.21] [code]
- [Ant Group] Auto-Rubric as Reward: From Implicit Preferences to Explicit Multimodal Generative Criteria [arXiv 2605.08] [code]
- [Meta] Rethinking Rubric Generation for Improving LLM Judge and Reward Modeling for Open-ended Tasks [arxiv 2602.05]
- [Alibaba Tongyi Lab] Auto-Rubric: Learning From Implicit Weights to Explicit Rubrics for Reward Modeling [arxiv 2510.17] [code]
(3) Filtering mechanisms
- [Meta] Training AI Co-Scientists Using Rubric Rewards [arxiv 2512.23]
- The cot encyclopedia: Analyzing, predicting, and controlling how a reasoning model will think [ICLR 2026][(https://openreview.net/forum?id=ugZKZ8vufv)]
- [DeepSeek] Inference-Time Scaling for Generalist Reward Modeling [arxiv 2504.02]
- [NVIDIA] RLBFF: Binary Flexible Feedback to bridge between Human Feedback & Verifiable Rewards [ICLR 2026]
(4) Rubric-first mechanisms
- [Ant Group] Reinforcement Learning with Rubric Anchors [arxiv 2508.12]
(5) Composable pipeline framework
- AutoChecklist: Composable Pipelines for Checklist Generation and Scoring with LLM-as-a-Judge [arXiv 2603.07] [Code]
Beyond the source of synthesis, another important dimension is whether rubrics remain fixed during RL optimization; accordingly, existing approaches can be divided into static and dynamic paradigms.
Static synthesis generates a fixed set of rubrics prior to the RL or relies exclusively on the query during the RL process, making them entirely independent of the evolving policy distribution or rollout behavior.
- [Li Auto Inc] RubricHub: A Comprehensive and Highly Discriminative Rubric Dataset via Automated Coarse-to-Fine Generation [arxiv 2601.08] [code]
- [Apple] RubiCap: Rubric-Guided Reinforcement Learning for Dense Image Captioning [arxiv 2603.09]
- [Zhipu AI] Chaining the Evidence: Robust Reinforcement Learning for Deep Search Agents with Citation-Aware Rubric Rewards [arxiv 2601.06] [code]
- [Apple] Checklists Are Better Than Reward Models For Aligning Language Models [NeurIPS 2025] [code]
- [Meta] Advancedif: Rubric-Based Benchmarking and Reinforcement Learning for Advancing LLM Instruction Following [arxiv 2511.10] [code]
Dynamic synthesis updates, extracts, or regenerates rubrics online according to the current policy distribution or rollout behavior.
- [Allen AI] DR Tulu: Reinforcement Learning with Evolving Rubrics for Deep Research [arxiv 2511.19] [code]
- [Google] RubricEM: Meta-RL with Rubric-guided Policy Decomposition beyond Verifiable Rewards [arxiv 2605.10]
- [Scale AI] Online Rubrics Elicitation from Pairwise Comparisons [arxiv 2510.07]
- [Meta] Compute as Teacher: Turning Inference Compute Into Reference-Free Supervision [NeurIPS 2025 Spotlight]
- [ByteDance Seed] Reinforcing Chain-of-Thought Reasoning with Self-Evolving Rubrics [arxiv 2602.10] [code]
- RLAC: Reinforcement Learning with Adversarial Critic for Free-Form Generation Tasks [arxiv 2511.01] [code]
Synthesized rubrics specify the criteria used to evaluate a response, action sequence, or decision trajectory.
- Reward and Guidance through Rubrics: Promoting Exploration to Improve Multi-Domain Reasoning [arxiv 2511.12]
- ACE-RL: Adaptive Constraint-Enhanced Reward for Long-form Generation Reinforcement Learning [arxiv 2509.04]
- [Allen AI] DR Tulu: Reinforcement Learning with Evolving Rubrics for Deep Research [arxiv 2511.19] [code]
- RubricRAG: Towards Interpretable and Reliable LLM Evaluation via Domain Knowledge Retrieval for Rubric Generation [arxiv 2603.20]
- [StepFun] Step-DeepResearch Technical Report [arxiv 2512.20] [code]
- [Zhipu AI] Chaining the Evidence: Robust Reinforcement Learning for Deep Search Agents with Citation-Aware Rubric Rewards [arxiv 2601.06] [code]
- Evaluating Legal Reasoning Traces with Legal Issue Tree Rubrics [arxiv 2512.01] [code]
Modeling in Rubric RMs involves two closely related goals: optimizing the generator to construct high-quality rubrics, and optimizing the judge to perform reliable rubric-guided judgment
(1) training a generator
- [Tencent] Learning Query-Specific Rubrics from Human Preferences for DeepResearch Report Generation [arxiv 2602.03]
- [Ant Group] OptimSyn: Influence-Guided Rubrics Optimization for Synthetic Data Generation [ICLR 2026]
- RubricRAG: Towards Interpretable and Reliable LLM Evaluation via Domain Knowledge Retrieval for Rubric Generation [arxiv 2603.20]
(2) training a judge
- [Microsoft] LLM-RUBRIC : A Multidimensional, Calibrated Approach to Automated Evaluation of Natural Language Texts [ACL 2024] [code]
- [ModelBest Inc] An Efficient Rubric-based Generative Verifier for Search-Augmented LLMs [arxiv 2510.14]
(3) training jointly
- Alternating Reinforcement Learning for Rubric-Based Reward Modeling in Non-Verifiable LLM Post-Training [arxiv 2602.01]
- [Tencent Hunyuan] DeltaRubric: Generative Multimodal Reward Modeling via Joint Planning and Verification [arXiv 202605.09] [code]
- RM-R1: Reward Modeling as Reasoning [ICLR 2026] [code]
- [DeepSeek] Inference-Time Scaling for Generalist Reward Modeling [arxiv 2504.02]
How to training generator and judge?
(1) Supervised Fine-Tuning
- OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignment [arxiv 2510.07]
- [Meta] Advancedif: Rubric-Based Benchmarking and Reinforcement Learning for Advancing LLM Instruction Following [arxiv 2511.10] [code]
- [MeiTuan] CDRRM: Contrast-Driven Rubric Generation for Reliable and Interpretable Reward Modeling [arxiv 2603.08] [code]
(2) Reinforcement Learning
- RLAC: Reinforcement Learning with Adversarial Critic for Free-Form Generation Tasks [arxiv 2511.01] [code]
- [Tencent] Learning Query-Specific Rubrics from Human Preferences for DeepResearch Report Generation [arxiv 2602.03]
- RubricRAG: Towards Interpretable and Reliable LLM Evaluation via Domain Knowledge Retrieval for Rubric Generation [arxiv 2603.20]
- RM-R1: Reward Modeling as Reasoning [ICLR 2026] [code]
- [ModelBest Inc] An Efficient Rubric-based Generative Verifier for Search-Augmented LLMs [arxiv 2510.14]
(3) Inject rubric knowledge into the judge
- Multidimensional Rubric-oriented Reward Model Learning via Geometric Projection Reference Constraints [arxiv 2511.16]
- [Google DeepMind] Robust Reward Modeling via Causal Rubrics [ICLR 2026]
Rubric Reasoning focuses on how multiple criteria are jointly considered by the judge to produce the final reward or supervision signal. This process is inherently challenging because the relationships among criteria can be complex: they may be complementary, partially overlapping, hierarchically dependent, or even conflicting.
Explicit aggregation maps criterion-based judgments over a selected rubric set into a final scalar value through an explicit aggregation function.
Linear Aggregation assumes independence between criteria
(1) Weighted aggregation
- [Scale AI] Rubrics as Rewards: Reinforcement Learning Beyond Verifiable domains [NeurIPS 2025]
- [Scale AI] Online Rubrics Elicitation from Pairwise Comparisons [arxiv 2510.07]
- [Meta] Rethinking Rubric Generation for Improving LLM Judge and Reward Modeling for Open-ended Tasks [arxiv 2602.05]
(2) Unweighted aggregation
- AUTORULE : Reasoning Chain-of-thought Extracted Rule-based Rewards Improve Preference Learning [arxiv 2506.15] [code]
- [Microsoft] RubricRL: Simple Generalizable Rewards for Text-to-Image generation [arxiv 2511.20]
(3) Aggregation between Rule RMs and Rubric RMs
- VERIF: Verification Engineering for Reinforcement Learning in Instruction Following [EMNLP 2025] [code]
- [Apple] Checklists Are Better Than Reward Models For Aligning Language Models [NeurIPS 2025] [code]
- [Lionrock AI Lab] Orcust: Stepwise-Feedback Reinforcement Learning for GUI Agent [arxiv 2509.17]
Structure-aware Aggregation captures non-linear trade-offs or hierarchical dependencies between criteria
- [Ant Group] Reinforcement Learning with Rubric Anchors [arxiv 2508.12]
- [Meta] Advancedif: Rubric-Based Benchmarking and Reinforcement Learning for Advancing LLM Instruction Following [arxiv 2511.10] [code]
- Reward and Guidance through Rubrics: Promoting Exploration to Improve Multi-Domain Reasoning [arxiv 2511.12]
By leveraging LLM in-context learning, Implicit Aggregation bypasses item-by-item scoring to directly output a final scalar signal through internal reasoning over the entire rubric
- [Scale AI] Rubrics as Rewards: Reinforcement Learning Beyond Verifiable domains [NeurIPS 2025]
- SALMON: SELF-ALIGNMENT WITH INSTRUCTABLE REWARD MODELS [ICLR 2024] [code]
- [Xiaohongshu Inc] Curing Miracle Steps in LLM Mathematical Reasoning with Rubric Rewards [arxiv 2510.07] [code]
- [Kimi] Kimi K2: Self-Critique Rubric Reward for Open-Ended Alignment [arxiv 2507.20]
- OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignment [arxiv 2510.07]
Compared with traditional data synthesis methods, rubrics enable more fine-grained quality control over both synthesized outputs and their intermediate construction processes via structured criteria
- [Li Auto Inc] RubricHub: A Comprehensive and Highly Discriminative Rubric Dataset via Automated Coarse-to-Fine Generation [arxiv 2601.08] [code]
- [Ant Group] OptimSyn: Influence-Guided Rubrics Optimization for Synthetic Data Generation [ICLR 2026]
- Kardia-R1: Unleashing LLMs to Reason toward Understanding and Empathy for Emotional Support via Rubric-as-Judge Reinforcement Learning [arxiv 2512.01] [code]
- [Google DeepMind] Robust Reward Modeling via Causal Rubrics [ICLR 2026]
- Configurable preference tuning with rubric-guided synthetic data [ICML 2025 Workshop] [code]
- ARISE: Agentic Rubric-Guided Iterative Survey Engine for Automated Scholarly Paper Generation [arxiv 2511.17] [code]
Compared with traditional outcome feedback, rubric-based supervision provides denser and more structured learning signals, allowing models to more precisely identify which behaviors should be reinforced
(1) Outcome-level
- [Tencent] Learning Query-Specific Rubrics from Human Preferences for DeepResearch Report Generation [arxiv 2602.03]
- [Allen AI] DR Tulu: Reinforcement Learning with Evolving Rubrics for Deep Research [arxiv 2511.19] [code]
- [Meta] Compute as Teacher: Turning Inference Compute Into Reference-Free Supervision [NeurIPS 2025 Spotlight]
- [Microsoft] RubricRL: Simple Generalizable Rewards for Text-to-Image generation [arxiv 2511.20]
- [DeepSeek] Deepseek-v4: Towards highly efficient million-token context intelligence [Tech Report]
- [DeepSeek] Inference-Time Scaling for Generalist Reward Modeling [arxiv 2504.02]
(2) Process-level
- [Zhipu AI] Chaining the Evidence: Robust Reinforcement Learning for Deep Search Agents with Citation-Aware Rubric Rewards [arxiv 2601.06] [code]
- Evaluating Legal Reasoning Traces with Legal Issue Tree Rubrics [arxiv 2512.01] [code]
- [ByteDance Seed] Reinforcing Chain-of-Thought Reasoning with Self-Evolving Rubrics [arxiv 2602.10] [code]
- [Xiaohongshu Inc] Curing Miracle Steps in LLM Mathematical Reasoning with Rubric Rewards [arxiv 2510.07] [code]
(3) Token-level
- [Alibaba] Rubrics to Tokens: Bridging Response-level Rubrics and Token-level Rewards in Instruction Following Tasks [arXiv 202604.02] [Code]
(4) Self-rewarding mechanism
- [Allen AI] EvoLM: Self-Evolving Language Models through Co-Evolved Discriminative Rubrics [arXiv 202605.03] [code]
- [Ant Group] Self-Rewarding Rubric-Based Reinforcement Learning for Open-Ended Reasoning [arXiv 202509.25]
(5) Rubric-guided exploration
- [Li Auto Inc] Breaking the Exploration Bottleneck: Rubric-Scaffolded Reinforcement Learning for General LLM Reasoning [arXiv 202508.16] [Code]
- [Google] RubricEM: Meta-RL with Rubric-guided Policy Decomposition beyond Verifiable Rewards [arxiv 2605.10]
- Reward and Guidance through Rubrics: Promoting Exploration to Improve Multi-Domain Reasoning [arxiv 2511.12]
- Experience is the Best Teacher: Motivating Effective Exploration in Reinforcement Learning for LLMs [arXiv 202603.20] [Code]
- Think-with-Rubrics: From External Evaluator to Internal Reasoning Guidance [arXiv 202605.07]
At inference time, Rubric RMs can serve as a verifier to directly improve policy outputs, extending its role beyond training supervision into inferencetime decision making
- [Tencent] Inference-Time Scaling of Verification: Self-Evolving Deep Research Agents via Test-Time Rubric-Guided Verification [arXiv 202601.15] [Code]
- [Scale AI] Agentic Rubrics as Contextual Verifiers for SWE Agents [arxiv 2601.04]
Rubrics are naturally applicable across a wide range of domains because many downstream tasks rely on implicit expert knowledge that cannot be easily captured by preference signals
Open-ended Chat
- QuRL: Rubrics As Judge For Open-Ended Question Answering [ICLR 26]
- [Ant Group] Auto-Rubric as Reward: From Implicit Preferences to Explicit Multimodal Generative Criteria [arXiv 2605.08] [code]
- [Scale AI] Rubrics as Rewards: Reinforcement Learning Beyond Verifiable domains [NeurIPS 2025]
- ACE-RL: Adaptive Constraint-Enhanced Reward for Long-form Generation Reinforcement Learning [arxiv 2509.04]
Medicine
- [OpenAI] HealthBench: Evaluating Large Language Models Towards Improved Human Health [arxiv 2505.08] [resource]
- [InfiX] InfiMed-ORBIT: Aligning LLMs on Open-Ended Complex Tasks via Rubric-Based Incremental Training [arxiv 2510.15] [code]
- Multidimensional Rubric-oriented Reward Model Learning via Geometric Projection Reference Constraints [arxiv 2511.16]
Law
- Evaluating Legal Reasoning Traces with Legal Issue Tree Rubrics [arxiv 2512.01] [code]
Science
- [Meta] Training AI Co-Scientists Using Rubric Rewards [arxiv 2512.23]
Mathematics
- [Xiaohongshu Inc] Curing Miracle Steps in LLM Mathematical Reasoning with Rubric Rewards [arxiv 2510.07] [code]
Coding
- [Scale AI] Agentic Rubrics as Contextual Verifiers for SWE Agents [arxiv 2601.04]
- AdaRubric: Task-Adaptive Rubrics for LLM Agent Evaluation [arxiv 2603.21] [code]
Multimodal
- [Ant Group] Auto-Rubric as Reward: From Implicit Preferences to Explicit Multimodal Generative Criteria [arXiv 2605.08] [code]
- [Microsoft] RubricRL: Simple Generalizable Rewards for Text-to-Image generation [arxiv 2511.20]
- [Apple] RubiCap: Rubric-Guided Reinforcement Learning for Dense Image Captioning [arxiv 2603.09]
- [Lionrock AI Lab] Orcust: Stepwise-Feedback Reinforcement Learning for GUI Agent [arxiv 2509.17]
- [Allen AI] When Rubrics Fail: Error Enumeration as Reward in Reference-Free RL Post-Training for Virtual Try-On [arXiv 202603.05]
- [Alibaba Qwen] Rationale matters: Learning transferable rubrics via proxy-guided critique for vlm reward models [arXiv 202603.16] [Code]
- Microverse: A preliminary exploration toward a micro-world simulation [ICLR 26] [Code]
- [Tencent Hunyuan] DeltaRubric: Generative Multimodal Reward Modeling via Joint Planning and Verification [arXiv 202605.09] [code]
DeepResearch
- [Google] RubricEM: Meta-RL with Rubric-guided Policy Decomposition beyond Verifiable Rewards [arxiv 2605.10]
- [Allen AI] DR Tulu: Reinforcement Learning with Evolving Rubrics for Deep Research [arxiv 2511.19] [code]
- [StepFun] Step-DeepResearch Technical Report [arxiv 2512.20] [code]
- [Zhipu AI] Chaining the Evidence: Robust Reinforcement Learning for Deep Search Agents with Citation-Aware Rubric Rewards [arxiv 2601.06] [code]
- [Tencent] Learning Query-Specific Rubrics from Human Preferences for DeepResearch Report Generation [arxiv 2602.03]
- [Tencent] Inference-Time Scaling of Verification: Self-Evolving Deep Research Agents via Test-Time Rubric-Guided Verification [arxiv 2605.09]
Fig 4. Overview of Dataset
we summarize the high-quality data resources currently used for training, including datasets with explicit rubric and source datasets for extract criteria
- [Li Auto Inc] RubricHub: A Comprehensive and Highly Discriminative Rubric Dataset via Automated Coarse-to-Fine Generation [arxiv 2601.08] [resource]
- [Apple] Checklists Are Better Than Reward Models For Aligning Language Models [NeurIPS 2025] [resource]
- OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignment [arxiv 2510.07] [resource]
- [Scale AI] Rubrics as Rewards: Reinforcement Learning Beyond Verifiable domains [NeurIPS 2025] [resource RaR-Medicine] [resource RaR-Science]
- Evaluating Legal Reasoning Traces with Legal Issue Tree Rubrics [arxiv 2512.01] [resource]
- [Alibaba Tongyi Lab] Auto-Rubric: Learning From Implicit Weights to Explicit Rubrics for Reward Modeling [arxiv 2510.17] [resource]
- [Allen AI] Wildchat: 1m chatGPT interaction logs in the wild [ICLR 2024] [resource]
- ULTRAFEEDBACK: Boosting language models with scaled AI feedback [ICML 2024] [resource]
- [Skywork AI] Skyworkreward: Bag of tricks for reward modeling in llms [arxiv 2410.18] [resource]
- [NVIDIA] Helpsteer3-preference: Open human-annotated preference data across diverse tasks and languages [NeurIPS 2025] [resource]
- [Allen AI] Magpie: Alignment data synthesis from scratch by prompting aligned LLMs with nothing [ICLR 2025] [resource]
- Megascience: Pushing the frontiers of post-training datasets for science reasoning [arxiv 2507.16] [resource]
- [Zhipu AI] Longwriter: Unleashing 10,000+ word generation from long context LLMs [ICLR 2025] [resource]
- Longwriterzero: Mastering ultra-long text generation via reinforcement learning [arxiv 2506.18] [resource]
- [Zhipu AI] LongAlign: A recipe for long context alignment of large language models [EMNLP 2024] [resource]
- LMSYS-chat-1m: A large-scale realworld LLM conversation dataset [ICLR 2024] [resource]
- Huatuogpt-o1, towards medical complex reasoning with llms [arxiv 2412.18] [resource]
- Rlaif-v: Opensource ai feedback leads to super gpt-4v trustworthiness [CVPR 2025] [resource]
- [Microsoft] Aligning large multimodal models with factually augmented RLHF [ACL 2024] [resource]
- [ByteDance] Lllava-critic: Learning to evaluate multimodal models [CVPR 2025] [resource]
- [KuaiShou] MM-RLHF: The next step forward in multimodal LLM alignment [ICML 2025] [resource]
- [Allen AI] Molmo and pixmo: Open weights and open data for stateof-the-art vision-language models [CVPR 2025] [resource]
- Densefusion-1m: Merging vision experts for comprehensive multimodal perception [NeurIPS 2024] [resource]
- VL-rethinker: Incentivizing selfreflection of vision-language models with reinforcement learning [NeurIPS 2025 Spotlight] [resource]
Fig 5. Overview of reward model benchmarks, grouped into rubric-based and general reward benchmarks, and compared by evaluation paradigm, domain, modality, and scale.
We review existing open-source benchmarks and evaluation methods for Rubric RMs evaluation from two perspectives: Rubric quality evaluates the model’s ability to generate accurate and useful rubrics, which directly measures the quality of rubrics. Rubric-guide judgment evaluates whether the generated rubric can support reliable reward modeling and downstream task judgment.
- [Tencent Hunyuan] RubricBench: Aligning Model-Generated Rubrics with Human Standards [arxiv 2603.01] [resource]
- [Ant Group] RUBRICEVAL: A Rubric-Level Meta-Evaluation Benchmark for LLM Judges in Instruction Following [arxiv 2603.25]
- [OpenAI] HealthBench: Evaluating Large Language Models Towards Improved Human Health [arxiv 2505.08] [resource]
- [OpenAI] PaperBench: Evaluating AI's Ability to Replicate AI Research [ICML 2025] [resource]
- [ByteDance Seed] Xpertbench: Expert Level Tasks with Rubrics-Based Evaluation [arxiv 2604.02] [resource]
- PresentBench: A Fine-Grained Rubric-Based Benchmark for Slide Generation [arXiv 2603.07] [resource]
- [NVIDIA] Profbench: Multidomain rubrics requiring professional knowledge to answer and judge [arXiv 2510.17] [resource]
- [Shanghai AI Lab] Generative judge for evaluating alignment [ICLR 2024] [resource]
- [Allen AI] RewardBench: Evaluating reward models for language modeling [NAACL 2025] [resource]
- Judging llm-as-a-judge with mtbench and chatbot arena [NeurIPS 2023] [resource]
- RMbench: Benchmarking reward models of language models with subtlety and style [ICLR 2025] [resource]
- Judgebench: A benchmark for evaluating LLM-based judges [ICLR 2025] [resource]
- [Meta] M-RewardBench: Evaluating reward models in multilingual settings [ACL 2025] [resource]
- RAG-RewardBench: Benchmarking reward models in retrieval augmented generation for preference alignment [ACL 2025] [resource]
- Evaluating robustness of reward models for mathematical reasoning [arXiv 2410.01] [resource]
- [Allen AI] Rewardbench 2: Advancing reward model evaluation [arXiv 2506.01] [resource]
- Rmb: Comprehensively benchmarking reward models in llm alignment [ICLR 2025] [resource]
- How to evaluate reward models for RLHF [ICLR 2025] [resource]
- [Allen AI] Vl-rewardbench: a challenging benchmark for vision-language generative reward models [CVPR 2025] [resource]
- [Meta] Multimodal rewardbench: Holistic evaluation of reward models for vision language models [arXiv 2502.14] [resource]
- MJ-bench: Is your multimodal reward model really a good judge for text-to-image generation? [NeurIPS 2025] [resource]