LongBench v2 / 66f95126bb02136c067c5070 / Compared with lets verify step by step, which of the following points is not…

Let’s Verify Step by Step
OpenAI
Abstract
In recent years, large language models have greatly improved in their
ability to perform complex multi-step reasoning.
However, even state-
of-the-art models still regularly produce logical mistakes. To train more
reliable models, we can turn either to outcome supervision, which provides
feedback for a final result, or process supervision, which provides feedback
for each intermediate reasoning step. Given the importance of training
reliable models, and given the high cost of human feedback, it is impor-
tant to carefully compare the both methods. Recent work has already
begun this comparison, but many questions still remain. We conduct our
own investigation, finding that process supervision significantly outper-
forms outcome supervision for training models to solve problems from the
challenging MATH dataset. Our process-supervised model solves 78% of
problems from a representative subset of the MATH test set. Additionally,
we show that active learning significantly improves the efficacy of process
supervision. To support related research, we also release PRM800K, the
complete dataset of 800,000 step-level human feedback labels used to train
our best reward model.
1
Introduction
Large language models are capable of solving tasks that require complex multi-
step reasoning by generating solutions in a step-by-step chain-of-thought format
(Nye et al., 2021; Wei et al., 2022; Kojima et al., 2022). However, even state-
of-the-art models are prone to producing falsehoods — they exhibit a tendency
to invent facts in moments of uncertainty (Bubeck et al., 2023). These hallu-
cinations (Maynez et al., 2020) are particularly problematic in domains that
require multi-step reasoning, since a single logical error is enough to derail a
much larger solution. Detecting and mitigating hallucinations is essential to
improve reasoning capabilities.
∗Primary authors. Correspondence to: Karl Cobbe <karl@openai.com>
1
arXiv:2305.20050v1  [cs.LG]  31 May 2023


One effective method involves training reward models to discriminate be-
tween desirable and undesirable outputs. The reward model can then be used
in a reinforcement learning pipeline (Ziegler et al., 2019; Stiennon et al., 2020;
Nakano et al., 2021; Ouyang et al., 2022) or to perform search via rejection sam-
pling (Nichols et al., 2020; Shen et al., 2021; Cobbe et al., 2021). While these
techniques are useful, the resulting system is only as reliable as the reward
model itself. It is therefore important that we study how to most effectively
train reliable reward models.
In closely related work, Uesato et al. (2022) describe two distinct meth-
ods for training reward models: outcome supervision and process supervision.
Outcome-supervised reward models (ORMs) are trained using only the final
result of the model’s chain-of-thought, while process-supervised reward models
(PRMs) receive feedback for each step in the chain-of-thought. There are com-
pelling reasons to favor process supervision. It provides more precise feedback,
since it specifies the exact location of any errors that occur. It also has sev-
eral advantages relevant to AI alignment: it is easier for humans to interpret,
and it more directly rewards models for following a human-endorsed chain-of-
thought. Within the domain of logical reasoning, models trained with outcome
supervision regularly use incorrect reasoning to reach the correct final answer
(Zelikman et al., 2022; Creswell et al., 2022).
Process supervision has been
shown to mitigate this misaligned behavior (Uesato et al., 2022).
Despite these advantages, Uesato et al. (2022) found that outcome supervi-
sion and process supervision led to similar final performance in the domain of
grade school math. We conduct our own detailed comparison of outcome and
process supervision, with three main differences: we use a more capable base
model, we use significantly more human feedback, and we train and test on the
more challenging MATH dataset (Hendrycks et al., 2021).
Our main contributions are as follows:
1. We show that process supervision can train much more reliable reward
models than outcome supervision. We use our state-of-the-art PRM to
solve 78.2% of problems from a representative subset of the MATH test
set.
2. We show that a large reward model can reliably approximate human su-
pervision for smaller reward models, and that it can be used to efficiently
conduct large-scale data collection ablations.
3. We show that active learning leads to a 2.6× improvement in the data
efficiency of process supervision.
4. We release our full process supervision dataset, PRM800K, to promote
related research.
2


2
Methods
We perform a comparison of outcome and process supervision, following a sim-
ilar methodology to Uesato et al. (2022). Outcome supervision can be provided
without humans, since all problems in the MATH dataset have automatically
checkable answers. In contrast, there is no simple way to automate process su-
pervision. We therefore rely on human data-labelers to provide process super-
vision, specifically by labelling the correctness of each step in model-generated
solutions.
We conduct experiments in two separate regimes: large-scale and small-
scale. Each has its own advantages, and they offer complimentary perspectives.
At large-scale, we finetune all models from GPT-4 (OpenAI, 2023). We focus
on advancing the state-of-the-art by training the most reliable ORM and PRM
possible. Unfortunately the training sets for these reward models are not directly
comparable, for reasons we will discuss in Section 3. These models are therefore
not ideal for making an apples-to-apples comparison of outcome and process
supervision. To address this flaw, we also train models at small-scale, where
we can conduct a more direct comparison. In order to remove our dependence
on costly human feedback, we use a large-scale model to supervise small-scale
model training. This setup enables us to conduct several important ablations
that would otherwise be infeasible.
2.1
Scope
At each model scale, we use a single fixed model to generate all solutions. We
call this model the generator. We do not attempt to improve the generator with
reinforcement learning (RL). When we discuss outcome and process supervision,
we are specifically referring to the supervision given to the reward model. We do
not discuss any supervision the generator would receive from the reward model
if trained with RL. Although finetuning the generator with RL is a natural next
step, it is intentionally not the focus of this work.
We instead focus exclusively on how to train the most reliable reward model
possible. We evaluate a reward model by its ability to perform best-of-N search
over uniformly sampled solutions from the generator. For each test problem we
select the solution ranked highest by the reward model, automatically grade it
based on its final answer, and report the fraction that are correct. A reward
model that is more reliable will select the correct solution more often.
2.2
Base Models
All large-scale models are finetuned from the base GPT-4 model (OpenAI, 2023).
This model has been pretrained solely to predict the next token; it has not been
pretrained with any Reinforcement Learning from Human Feedback (RLHF)
(Christiano et al., 2017). The small-scale base models are similar in design to
GPT-4, but they were pretrained with roughly 200 times less compute. As an
additional pretraining step, we finetune all models on a dataset of roughly 1.5B
3


Figure 1: A screenshot of the interface used to collect feedback for each step in
a solution.
math-relevant tokens, which we call MathMix.
Similar to Lewkowycz et al.
(2022), we find that this improves the model’s mathematical reasoning capabil-
ities. Details on how this dataset was constructed can be found in Appendix A.
2.3
Generator
To make parsing individual steps easier, we train the generator to produce
solutions in a newline delimited step-by-step format. Specifically, we few-shot
generate solutions to MATH training problems, filter to those that reach the
correct final answer, and finetune the base model on this dataset for a single
epoch. This step is not intended to teach the generator new skills; it is intended
only to teach the generator to produce solutions in the desired format.
2.4
Data Collection
To collect process supervision data, we present human data-labelers with step-
by-step solutions to MATH problems sampled by the large-scale generator.
Their task is to assign each step in the solution a label of positive, negative,
or neutral, as shown in Figure 1. A positive label indicates that the step is cor-
rect and reasonable. A negative label indicates that the step is either incorrect
or unreasonable. A neutral label indicates ambiguity. In practice, a step may
be labelled neutral if it is subtly misleading, or if it is a poor suggestion that
is technically still valid. We permit neutral labels since this allows us to defer
the decision about how to handle ambiguity: at test time, we can treat neutral
labels as either positive or negative. A more detailed description of the labelling
instructions is provided in Appendix D.
We label solutions exclusively from the large-scale generator in order to
maximize the value of our limited human-data resource. We refer to the en-
tire dataset of step-level labels collected as PRM800K. The PRM800K training
set contains 800K step-level labels across 75K solutions to 12K problems. To
4


minimize overfitting, we include data from 4.5K MATH test problems in the
PRM800K training set, and we therefore evaluate our models only on the re-
maining 500 MATH test problems. More details about this test set can be found
in Appendix C.
During data collection, we must decide which solutions to surface to data-
labelers. The most straightforward strategy is to uniformly surface solutions
produced by the generator. However, if we surface solutions that make obvious
errors, the human feedback we get is less valuable. We would prefer to surface
solutions that are more likely to fool our best reward model. To that end, we at-
tempt to strategically select which solutions to show data-labelers. Specifically,
we choose to surface convincing wrong-answer solutions. We use the term con-
vincing to refer to solutions that are rated highly by our current best PRM, and
we use wrong-answer to refer to solutions that reach an incorrect final answer.
We use this slightly verbose phrasing to emphasize the fact that correctness is
determined solely by checking the final answer, a process which occasionally
leads to misgraded solutions. We expect to gain more information from labeling
convincing wrong-answer solutions, since we know the PRM is mistaken about
at least one step in each such solution.
In addition to using this selection strategy, we also iteratively re-train our
PRM using the latest data at several points in the data collection process. At
each iteration, we generate N solutions per problem and surface only the top K
most convincing wrong-answer solutions to data-labelers. We experiment with
either applying this top-K filtering at a problem level (K solutions per problem)
or globally across the dataset (K solutions in total, unequally distributed among
problems). Since the data collection process is expensive, it was not feasible
to conduct at-scale ablations of these decisions. However, we perform several
surrogate ablations in Section 4, using our largest PRM as a labelling oracle for
a smaller PRM. More details about data collection can be found in Appendix B.
2.5
Outcome-supervised Reward Models (ORMs)
We train ORMs following a similar methodology to Cobbe et al. (2021). We
uniformly sample a fixed number of solutions per problem from the generator,
and we train the ORM to predict whether each solution is correct or incorrect.
In practice, we usually determine correctness by automatically checking the
final answer, but in principle these labels could be provided by humans. At test
time, we use the ORM’s prediction at the final token as the overall score for the
solution. We note the automatic grading used to determine ORM targets is not
perfectly reliable: false positives solutions that reach the correct answer with
incorrect reasoning will be misgraded.
We discuss additional ORM training
details in Appendix E.
2.6
Process-supervised Reward Models (PRMs)
We train PRMs to predict the correctness of each step after the last token in
each step. This prediction takes the form of a single token, and we maximize the
5


Figure 2: Two solutions to the same problem, graded by the PRM. The solution
on the left is correct while the solution on the right is incorrect.
A green
background indicates a high PRM score, and a red background indicates a low
score. The PRM correctly identifies the mistake in the incorrect solution.
log-likelihood of these target tokens during training. The PRM can therefore
be trained in a standard language model pipeline without any special accom-
modations. To determine the step-level predictions at test time, it suffices to
perform a single PRM forward pass over the whole solution. We visualize large-
scale PRM scores for two different solutions in Figure 2. To compare multiple
solutions, it is necessary to compute a single score for each solution. This is an
important but straightforward detail: we define the PRM score for a solution to
be the probability that every step is correct under the PRM. We implement this
as the product of the correctness probabilities for each step. We describe other
possible scoring strategies and additional PRM training details in Appendix F.
When we provide process supervision, we deliberately choose to supervise
only up to the first incorrect step. This makes the comparison between out-
come and process supervision more straightforward. For correct solutions, both
methods provide the same information, namely that every step is correct. For
incorrect solutions, both methods reveal the existence of at least one mistake,
and process supervision additionally reveals the precise location of that mistake.
If we were to provide additional process supervision beyond the first mistake,
then process supervision would have an even greater information advantage.
This decision also keeps the labelling cost similar for humans: without relying
on an easy-to-check final answer, determining the correctness of a solution is
equivalent to identifying its first mistake. While most MATH problems do have
easy-to-check final answers, we expect this to not remain true in more complex
domains.
6


ORM
PRM
Majority Voting
% Solved (Best-of-1860)
72.4
78.2
69.6
101
102
103
N = number of solutions per problem
62
64
66
68
70
72
74
76
78
% Problems Solved (Best-of-N)
Process-Supervised RM
Outcome-Supervised RM
Majority Voting
Figure 3: A comparison of outcome-supervised and process-supervised reward
models, evaluated by their ability to search over many test solutions. Majority
voting is shown as a strong baseline. For N ≤1000, we visualize the variance
across many subsamples of the 1860 solutions we generated in total per problem.
3
Large-scale Supervision
We train the large-scale PRM using the step-level labels in PRM800K. To ensure
the large-scale ORM baseline is as strong as possible, we train on 100 uniform
samples per problem from the generator. This means the ORM training set has
no overlap with PRM800K, and it is an order of magnitude larger. Although
these two training sets are not directly comparable, each represents our best
attempt to advance the state-of-the-art with each form of supervision. We note
that training the ORM solely on PRM800K solutions would be problematic,
since our active learning strategy has heavily biased the dataset towards wrong-
answer solutions. We did explore training the ORM on a superset of PRM800K
solutions, by mixing in uniformly sampled solutions, but we found that this did
not improve ORM performance.
Figure 3 shows how the best-of-N performance of each reward model varies
as a function of N. Since majority voting is known to be a strong baseline (Wang
et al., 2022; Lewkowycz et al., 2022), we also include this method as a point of
comparison. While the ORM performs slightly better than the majority voting
baseline, the PRM strongly outperforms both. Not only does the PRM reach
higher performance for all values of N, but the performance gap widens as N
increases. This indicates that the PRM is more effective than both the ORM and
majority voting at searching over a large number of model-generated solutions.
7


100
101
102
Number of solutions labelled per problem
25
30
35
40
45
50
55
60
% Problems Solved (Best-of-500)
PRM + Active Learning
PRM (PRMlarge supervised)
ORM (PRMlarge supervised)
ORM (final-answer supervised)
(a)
Four
series
of
reward
models
trained using different data collection
strategies, compared across training
sets of varying sizes.
100
101
102
103
N = number of solutions per problem
20
25
30
35
40
45
50
55
60
% Problems Solved (Best-of-N)
PRM (PRMlarge supervised)
ORM (PRMlarge supervised)
ORM (final-answer supervised)
(b) Three reward models trained on
200 samples/problem using different
forms of supervision, compared across
many test-time compute budgets.
Figure 4: A comparison of different forms of outcome and process supervision.
Mean and standard deviation is shown across three seeds.
We experimented with using RM-weighted voting (Li et al., 2022; Uesato et al.,
2022) to combine the benefits of the PRM and majority voting, but this did not
noticeably improve performance. We use a specific subset of the MATH test set
for evaluation, which we describe in Appendix C. We further break down these
results by problem difficulty in Appendix G.
4
Small-scale Synthetic Supervision
We find that the PRM outperforms the ORM at large-scale, but this result alone
paints an incomplete picture. To better compare outcome and process supervi-
sion, there are two confounding factors that must be isolated. First, the training
sets for the ORM and the PRM are not directly comparable: the PRM training
set was constructed using active learning, is biased towards answer-incorrect
solutions, and is an order of magnitude smaller. Second, the final-answer grad-
ing will provide positive labels to spurious solutions that reach the correct final
answer despite incorrect reasoning. This could damage ORM performance, an
effect we may or may not want to attribute to outcome supervision more gen-
erally.
Due to the high cost of collecting human feedback, we cannot easily ablate
these factors using human labelers. We instead perform the relevant ablations
by using the large-scale PRM to supervise smaller models. This setup enables
us to simulate a large amount of data collection at a modest cost.
For the
remainder of this section, we refer to the large-scale PRM from Section 3 as
PRMlarge.
8


4.1
Process vs Outcome Supervision
We now conduct a direct comparison of outcome and process supervision. We
first sample between 1 and 200 solutions per problem from a small-scale genera-
tor. For each dataset, we provide three forms of supervision: process supervision
from PRMlarge, outcome supervision from PRMlarge, and outcome supervision
from final-answer checking.
The choice of supervision is the only difference
between these three series of reward models, which are otherwise trained on
identical datasets. See Appendix H for more details about how PRMlarge is
used for outcome and process supervision.
In Figure 4a, we evaluate each reward model by its best-of-500 selection. We
see that process supervision significantly outperforms both forms of outcome
supervision at all data collection scales.
In Figure 4b, we evaluate the best
reward model from each series by its best-of-N performance across different
values of N. We see that using PRMlarge for outcome supervision is noticeably
more effective than final-answer checking. This can be explained by the fact
that PRMlarge provides better supervision for solutions that reach the correct
final answer using incorrect reasoning.
It is not clear whether supervision by PRMlarge or by final-answer checking
represents the more appropriate outcome supervision baseline.
While final-
answer supervision is more explicitly outcome based, its main weakness — the
existence of false positives — is arguably over-emphasized in the MATH dataset.
Outcome supervision by PRMlarge better represents outcome supervision in do-
mains that are less susceptible to false positives. We consider outcome supervi-
sion by PRMlarge to be the more relevant baseline, but we encourage the reader
to draw their own conclusions.
4.2
Active Learning
Finally, we investigate the impact of active learning. We train a small-scale
reward model, PRMselector, on a single sample from each problem, and we use
this model to score 1000 samples per problem. To train each of our larger re-
ward models, we select N samples per problem such that 80% are the most
convincing (according to PRMselector) wrong-answer samples, and 20% are the
most convincing samples that remain (right- or wrong-answer). We score the
selected samples with PRMlarge and train on those scores. This process ensures
that all samples are relatively convincing under PRMselector, that a large frac-
tion are known to contain at least one mistake, and that our overall dataset
is not too heavily biased toward wrong-answer solutions. Performance of this
data labelling scheme is shown in Figure 4a. By comparing the slopes of the
line of best fit with and without active learning, we estimate that this form
of active learning is approximately 2.6x more data efficient than uniform data
labelling. We note that the model trained on the largest active learning dataset
(200 samples per problem) appears to slightly underperform the expected trend
line. Our best explanation for this observation is that 200 samples represents
a significant fraction of the overall selection pool (1000 samples) and that this
9


ORM
PRM
Majority Vote
# Problems
AP Calculus
68.9%
86.7%
80.0%
45
AP Chemistry
68.9%
80.0%
71.7%
60
AP Physics
77.8%
86.7%
82.2%
45
AMC10/12
49.1%
53.2%
32.8%
84
Aggregate
63.8%
72.9%
61.3%
234
Table 1: We measure out-of-distribution generalization using recent STEM tests.
We evaluate the outcome-supervised RM, the process-supervised RM, and ma-
jority voting using 100 test samples per problem.
relative lack of diversity limits the possible upside from active learning.
We also performed a preliminary investigation into the impact of iteratively
retraining PRMselector throughout data collection. Between iterations, we re-
trained PRMselector using all currently labeled data. Unfortunately, we observed
instability in this process which we were unable to diagnose.
The resulting
reward models performed no better than the models described above. We expect
some form of iterative retraining to be beneficial in active learning, but we
currently have no concrete evidence to support this claim. We consider this a
compelling direction for future research.
5
OOD Generalization
To get some measure of out-of-distribution generalization, we evaluate our large-
scale ORM and PRM on a held-out set of 224 STEM questions, pulled from the
most recent AP Physics, AP Calculus, AP Chemistry, AMC10, and AMC12 ex-
ams. Since these tests were released after the pre-training dataset was compiled,
we can have high confidence that the model has not seen these problems. We
report the best-of-100 performance of the ORM, PRM and majority voting in
Table 1. We observe results similar to those in Section 3: the PRM outperforms
both the ORM and majority voting. This shows us that the PRM can tolerate
a modest amount of distribution shift and that its strong performance holds up
on fresh test questions.
6
Discussion
6.1
Credit Assignment
One clear advantage of process supervision is that it provides more precise
feedback than outcome supervision.
A reward model trained with outcome
supervision faces a difficult credit-assignment task — to generalize well, it must
determine where an incorrect solution went wrong. This is particularly difficult
for hard problems: most model-generated solutions contain an error somewhere,
so the marginal value of a negative label from outcome supervision is low. In
10


contrast, process supervision provides a richer signal: it specifies both how
many of the first steps were in fact correct, as well as the precise location of
the incorrect step. Process supervision makes credit assignment easier, and we
believe that this explains its strong performance.
6.2
Alignment Impact
Process supervision has several advantages over outcome supervision related
to AI alignment. Process supervision is more likely to produce interpretable
reasoning, since it encourages models to follow a process endorsed by humans.
Process supervision is also inherently safer: it directly rewards an aligned chain-
of-thought rather than relying on outcomes as a proxy for aligned behavior
(Stuhlm¨
uller and Byun, 2022). In contrast, outcome supervision is harder to
scrutinize, and the preferences conveyed are less precise.
In the worst case,
the use of outcomes as an imperfect proxy could lead to models that become
misaligned after learning to exploit the reward signal (Uesato et al., 2022; Cotra,
2022; Everitt et al., 2017).
In some cases, safer methods for AI systems can lead to reduced performance
(Ouyang et al., 2022; Askell et al., 2021), a cost which is known as an alignment
tax. In general, any alignment tax may hinder the adoption of alignment meth-
ods, due to pressure to deploy the most capable model. Our results show that
process supervision in fact incurs a negative alignment tax. This could lead to
increased adoption of process supervision, which we believe would have positive
alignment side-effects. It is unknown how broadly these results will generalize
beyond the domain of math, and we consider it important for future work to
explore the impact of process supervision in other domains.
6.3
Test Set Contamination
The test set of the MATH dataset contains problems that are discussed in
several online venues, and it is likely that some of these problems appear in
the pretraining dataset for our models. We attempted to remove all MATH
problems from our MathMix dataset using string-matching heuristics, but since
humans can post hard-to-detect rephrasings of a problem online, it is difficult
to make any strong guarantees about the overlap between MathMix and the
MATH dataset.
In our experience inspecting model-generated solutions, we saw no clear signs
of our models memorizing MATH problems. However, it is impossible to rule
out subtle forms of memorization that would slip past manual inspection, and
it is still possible that some degree of contamination has slightly inflated our
performance on the MATH test set. Even in that case, we would expect any
contamination to manifest similarly across all methods, and that the relative
comparisons made throughout this work would remain mostly unaffected.
We also note that the PRM regularly surfaces correct solutions to MATH
problems that have a low single-digit percentage solve-rate under the genera-
tor, some examples of which can be seen in Appendix I. The generator’s low
11


solve-rate is an additional indication that it has not encountered such problems
via test set contamination. Our generalization results from Section 5 further
strengthen our claim that test set contamination has not significantly impacted
this work, since we observe qualitatively similar results on problems that are
guaranteed to be uncontaminated.
7
Related Work
7.1
Outcome vs Process Supervision
In work closely related to our own, Uesato et al. (2022) compare the impact
of outcome and process supervision in the domain of grade school math. They
found that both methods led to similar final-answer error rates, and that process
supervision achieved those results with less data. While our core methodology is
very similar, there are three main details that differ. First, we use a more capable
model to collect PRM800K dataset and to perform our large-scale experiments.
However, our small-scale results in Section 4 suggest that large-scale models are
not necessary to observe benefits from process supervision. Second, we evaluate
on the MATH dataset, which is significantly more challenging than GSM8K.
Third, we collect a much larger quantity of process supervision data.
On the surface, the results from Uesato et al. (2022) may seem to conflict
with our claim that process supervision leads to better performance. However,
we believe the apparent conflict can be explained by the difference in the scale
of the supervision. The data scaling trend in Figure 4a suggests that a small
amount of process supervision and a large amount of outcome supervision do
in fact lead to similar performance, consistent with the results from Uesato
et al. (2022).
The trend also shows that process supervision beats outcome
supervision when scaled up, even when judged based solely on outcomes. This
is consistent with our results in Section 3.
We believe these results make a
strong case for using process supervision.
7.2
Synthetic Supervision
Similar to our work in Section 4, Gao et al. (2022) use a large reward model to
supervise the training of smaller models. They study the over-optimization that
occurs during RLHF, with experiments that require large quantities of human
preference data. To work around this challenge, they use a gold-standard reward
model to replace human feedback. Our use of a large-scale reward model to
supervise smaller reward models shares similarities with their approach.
7.3
Natural Language Reasoning
Several recent studies that have examined the reasoning ability of large language
models are implicitly relevant to our work. Lewkowycz et al. (2022) showed that
finetuning models on a large corpus of technical content led to significantly im-
proved performance on MATH. Wang et al. (2022) showed that self-consistency
12


leads to remarkably strong performance on many reasoning benchmarks, no-
tably without requiring any additional finetuning. Wei et al. (2022) and Nye
et al. (2021) demonstrate the importance of explicitly performing intermediate
reasoning steps via a chain of thought or a scratchpad in order to solve tasks
that require multi-step reasoning. Kojima et al. (2022) show that models are
able to perform this behavior zero-shot, conditioned only on a simple prompt.
8
Conclusion
We have shown that process supervision can be used to train much more reliable
reward models than outcome supervision in the domain of mathematical rea-
soning. We have also shown that active learning can be used to lower the cost of
human data collection by surfacing only the most valuable model completions
for human feedback. We release PRM800K, the full dataset of human feedback
used to train our state-of-the-art reward model, with the hope that removing
this significant barrier to entry will catalyze related research on the alignment of
large language models. We believe that process supervision is currently under-
explored, and we are excited for future work to more deeply investigate the
extent to which these methods generalize.
Acknowledgements
We thank Joshua Achiam, Mark Chen, Jonathan Gordon, Dan Hendrycks,
Lukasz Kaiser, Oleg Murk, Ben Sokolowsky, Francis Song, and Jonathan Uesato
for valuable feedback and thoughtful discussions; Giambattista Parascandolo
and Daniel Selsam for their contributions to the MathMix dataset; Jonathan
Ward for contributing to the data collection interface; Wojciech Zaremba for en-
couraging us to scale up data collection; Peter Hoeschele and Aris Kostantinidis
for supporting our data collection; the research acceleration and supercomput-
ing teams at OpenAI for providing infrastructure support; and the team at Scale
and the many data-labelers who created PRM800K.
References
A. Askell, Y. Bai, A. Chen, D. Drain, D. Ganguli, T. Henighan, A. Jones,
N. Joseph, B. Mann, N. DasSarma, et al. A general language assistant as a
laboratory for alignment. arXiv preprint arXiv:2112.00861, 2021.
S. Bubeck, V. Chandrasekaran, R. Eldan, J. Gehrke, E. Horvitz, E. Kamar,
P. Lee, Y. T. Lee, Y. Li, S. Lundberg, et al.
Sparks of artificial general
intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712,
2023.
13


P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei.
Deep reinforcement learning from human preferences.
Advances in neural
information processing systems, 30, 2017.
K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert,
J. Tworek, J. Hilton, R. Nakano, et al. Training verifiers to solve math word
problems. arXiv preprint arXiv:2110.14168, 2021.
A.
Cotra.
Without
specific
countermeasures,
the
easi-
est
path
to
transformative
AI
likely
leads
to
AI
takeover.
https://www.alignmentforum.org/posts/pRkFkzwKZ2zfa3R6H/
without-specific-countermeasures-the-easiest-path-to, 2022.
A. Creswell, M. Shanahan, and I. Higgins.
Selection-inference:
Exploiting
large language models for interpretable logical reasoning.
arXiv preprint
arXiv:2205.09712, 2022.
T. Everitt, V. Krakovna, L. Orseau, M. Hutter, and S. Legg. Reinforcement
learning with a corrupted reward channel. arXiv preprint arXiv:1705.08417,
2017.
L. Gao, J. Schulman, and J. Hilton. Scaling laws for reward model overopti-
mization. arXiv preprint arXiv:2210.10760, 2022.
D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song,
and J. Steinhardt. Measuring mathematical problem solving with the math
dataset. arXiv preprint arXiv:2103.03874, 2021.
T. Kojima, S. S. Gu, M. Reid, Y. Matsuo, and Y. Iwasawa. Large language
models are zero-shot reasoners. arXiv preprint arXiv:2205.11916, 2022.
A. Lewkowycz, A. Andreassen, D. Dohan, E. Dyer, H. Michalewski, V. Ra-
masesh, A. Slone, C. Anil, I. Schlag, T. Gutman-Solo, et al.
Solving
quantitative reasoning problems with language models.
arXiv preprint
arXiv:2206.14858, 2022.
Y. Li, Z. Lin, S. Zhang, Q. Fu, B. Chen, J.-G. Lou, and W. Chen.
On
the advance of making language models better reasoners.
arXiv preprint
arXiv:2206.02336, 2022.
J. Maynez, S. Narayan, B. Bohnet, and R. McDonald.
On faithfulness and
factuality in abstractive summarization.
arXiv preprint arXiv:2005.00661,
2020.
R. Nakano, J. Hilton, S. Balaji, J. Wu, L. Ouyang, C. Kim, C. Hesse, S. Jain,
V. Kosaraju, W. Saunders, et al.
Webgpt:
Browser-assisted question-
answering with human feedback. arXiv preprint arXiv:2112.09332, 2021.
E. Nichols, L. Gao, and R. Gomez. Collaborative storytelling with large-scale
neural language models. In Proceedings of the 13th ACM SIGGRAPH Con-
ference on Motion, Interaction and Games, pages 1–10, 2020.
14


M. Nye, A. J. Andreassen, G. Gur-Ari, H. Michalewski, J. Austin, D. Bieber,
D. Dohan, A. Lewkowycz, M. Bosma, D. Luan, et al.
Show your work:
Scratchpads for intermediate computation with language models.
arXiv
preprint arXiv:2112.00114, 2021.
OpenAI. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin,
C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. Training language models to
follow instructions with human feedback. arXiv preprint arXiv:2203.02155,
2022.
J. Shen, Y. Yin, L. Li, L. Shang, X. Jiang, M. Zhang, and Q. Liu. Generate
& rank: A multi-task framework for math word problems. arXiv preprint
arXiv:2109.03034, 2021.
N. Stiennon, L. Ouyang, J. Wu, D. Ziegler, R. Lowe, C. Voss, A. Radford,
D. Amodei, and P. F. Christiano. Learning to summarize with human feed-
back.
Advances in Neural Information Processing Systems, 33:3008–3021,
2020.
A. Stuhlm¨
uller and J. Byun. Supervise process, not outcomes. https://ought.
org/updates/2022-04-06-process, 2022.
J. Uesato, N. Kushman, R. Kumar, F. Song, N. Siegel, L. Wang, A. Creswell,
G. Irving, and I. Higgins.
Solving math word problems with process-and
outcome-based feedback. arXiv preprint arXiv:2211.14275, 2022.
X. Wang, J. Wei, D. Schuurmans, Q. Le, E. Chi, and D. Zhou. Self-consistency
improves chain of thought reasoning in language models.
arXiv preprint
arXiv:2203.11171, 2022.
J. Wei, X. Wang, D. Schuurmans, M. Bosma, E. Chi, Q. Le, and D. Zhou.
Chain of thought prompting elicits reasoning in large language models. arXiv
preprint arXiv:2201.11903, 2022.
E. Zelikman, Y. Wu, J. Mu, and N. Goodman. Star: Bootstrapping reason-
ing with reasoning. Advances in Neural Information Processing Systems, 35:
15476–15488, 2022.
D. M. Ziegler, N. Stiennon, J. Wu, T. B. Brown, A. Radford, D. Amodei,
P. Christiano, and G. Irving. Fine-tuning language models from human pref-
erences. arXiv preprint arXiv:1909.08593, 2019.
15


A
MathMix
Similar to Lewkowycz et al. (2022) we construct a large-scale dataset of high-
quality math-relevant tokens for use in a lightweight pretraining stage, before
finetuning on comparably smaller datasets like MATH and PRM800K. This
dataset, which we call MathMix, has two main differences compared to the one
used to train Minerva. First, it is smaller and more aggressively filtered to high-
quality math problem-solving content, and second, it does not explicitly mix in
general language data.
Minerva was trained on 38.5B tokens of arXiv documents and webscrape
pages with LaTeX content, while MathMix consists of a smaller set of 1.5B
tokens containing individual math problems and their solutions, free-form text
discussing math problems and concepts, and synthetic data (Table 2). While
Minerva was pretrained on a dataset with 5% general natural language data,
we chose not to mix in any natural language data explicitly, primarily because
MathMix already contains plenty of natural language data.
Data type
Token count
Present in pretraining?
Math problems and solutions
∼275M
No
Free-form math discussion text (1)
∼430M
No
Free-form math discussion text (2)
∼450M
Yes
Synthetic data (1)
∼30M
No
Synthetic data (2)
∼100M
Yes
Critiques grading data
∼500M
No
Table 2: MathMix dataset components.
Note that when training smaller models, as in Section 4, we use a slightly
smaller variant of MathMix that excludes the critiques data and only consists of
1B tokens. For our large models experiments, we train on MathMix for roughly
3B tokens (2 epochs). For our small models experiments, we train for 6 epochs
(roughly 6.6B tokens).
We apply a set of decontamination checks on MathMix against the test
split of the MATH dataset, including stripping out LaTeX and searching for
matching n-grams, but we can make no strong guarantees on the efficacy of this
decontamination. As discussed in Section 6.3, we would not expect the relative
comparisons made throughout this work to be significantly impacted by test set
contamination.
16


B
PRM800K
We collected 1,085,590 step-level labels over 101,599 solution samples.
We
present the whole unfiltered dataset as PRM800K. During training we discard
labels used for quality control, as well as any step-level labels for which the
labeler was unable to complete the task. The filtered dataset contains about
800,000 step-level labels over 75,000 solutions. The full PRM800K dataset is
available at https://github.com/openai/prm800k.
The data collection was split into two separate phases. In phase 1, we col-
lected labels for multiple alternative completions at each step of a solution. This
seeded our dataset but was cumbersome—for many steps the alternatives were
repetitive, and we found labelers spent a lot of time supervising long uninter-
esting solutions. As a result, the step-level labels we collected in this phase are
more repetitive than those collected later. In total, phase 1 represents about
5% of PRM800K, or about 40,000 step-level labels.
The majority of our labels were collected as part of phase 2, during which we
scaled up and streamlined the data collection process. Phase 2 data collection
is split into 10 generations. For each generation, we sample N solutions per
problem from the generator.
We rank these solutions with our current best
PRM and surface the highest scoring wrong-answer solutions to our labelers.
We retrain this PRM between each generation using all the latest data. This
active learning strategy changes the balance of our data considerably. Though
we sometimes surfaced correct solutions (either by manually injecting correct
solutions or because of errors in our automatic grading), the vast majority of the
labels we collected in this phase are for incorrect solutions. Table 3 breaks down
the balance of correct/incorrect steps and solutions between the different phases
of data collection. Though we mostly collected labels on incorrect solutions, we
still collected many labels for correct individual steps. In fact, our small-scale
ablations in Section 4.2 suggest that this active learning strategy, which favors
labelling high-scoring wrong-answer solutions, improves performance despite the
resulting imbalance in the dataset.
phase 1
phase 2
combined
% end in correct solution
85.1
13.2
14.2
% correct steps
58.6
74.1
73.1
Table 3: Distribution of positive/negative steps/solutions.
Some of our phase 2 questions are intended for quality control. For a quality
control question, researchers mark which steps are reasonable to label as in-
correct. Then we assess that labelers are able to consistently mark those steps
as incorrect. Prior to starting on phase 2, we required all labelers to label 30
quality control questions. This served as a screening test, and we only admitted
labelers that agreed with our gold labels at least 75% of the time.
We then designated 10-20 problems per generation as additional quality
control questions, and we randomly served them to labelers as they worked
17


through the task.
We used the results of this continuous quality control to
remove labelers whose quality slipped too far, as well as to prepare educational
material on common mistakes in order to improve labeler alignment with our
instructions.
C
Evaluation
As we scaled up the project, we began having to collect labels on multiple
solutions for the same training problem. In order to avoid the risk of over-fitting
on the 7,500 MATH training problems, we expanded the training set to include
4,500 MATH test split problems. We therefore evaluate our models only on the
remaining 500 held-out problems. We selected these 500 test problems uniformly
at random. In Figure 5, we show that the distribution of difficulty levels and
subjects in this subset is representative of the MATH test set as a whole. The
specific test set we used can be found at https://github.com/openai/prm800k.
We leave it for future work to explore how many distinct training problems are
actually necessary, and how quickly our methods overfit to the training set.
Figure 5: Two histograms comparing the distribution of problem difficulty levels
and subjects in both the original MATH test set and in our 500 problem test
subset.
18


D
Labelling Instructions
Labelers were tasked to look at steps in a solution and label each one as posi-
tive, negative, or neutral. A step is considered neutral if it is appropriate in
context, reasonable, correct, and contains only computations that can be veri-
fied easily. A step is positive if it is neutral and also progresses towards the
solution. All other steps are considered negative. Labelers were not given ref-
erence solutions, but they were given the ground truth final answers. We chose
not to provide reference solutions to avoid biasing them towards one particular
path to the solution. We chose to provide ground truth final answers since this
information can sometimes help labelers resolve their own misunderstandings.
In phase 1, labelers were permitted to enter their own steps in the case that
all candidate steps were negative. Then the solution would progress from a
randomly selected positive step (or neutral if their were no positive ones).
This often resulted in trajectories that got stuck in endless sequences of neutral
steps that said reasonable things but made frustratingly slow progress towards a
solution or negative steps that needed constant human supervision. In phase 2,
we pre-generate whole solutions and end the task as soon as the first negative
step is encountered.
The full instructions given to labelers can be found at
https://github.com/openai/prm800k/tree/main/prm800k/instructions.
E
ORM Training Details
We train outcome-supervised reward models in the same manner as token-level
verifiers from Cobbe et al. (2021), with a few subtle differences to hyperparam-
eters. In particular, we only train for a single epoch on each dataset of model
samples and reward model labels, without dropout, and without jointly learn-
ing a language modeling objective. We find that performance is not sensitive to
most other hyperparameters, within a reasonable range.
To collect model samples, we simply sample uniformly from the generator at
a temperature of 1.0 without applying any rebalancing of positives or negatives.
At training time, the reward model makes predictions for every token in the
context. The target for each token in a solution is the same, based on whether
the solution is labelled correct or incorrect. At test time, we simply use the
score of the final token in the completion as the overall score of the solution.
We note that this setup is identical to the way token-level verifiers were trained
in Cobbe et al. (2021).
19


F
PRM Details
F.1
Training
We train our PRMs by fine-tuning the MathMix model to predict the probability
of positive, negative, and neutral labels given a solution prefix ending in one of
our labeled steps. We sweep over hyperparameters using a dataset containing
the first ∼10% of PRM800K. Fine-tuning an LLM from its ordinary language
modeling task to a classification task like this is a large distribution shift, and
we found low learning rates were important to stable PRM training.
All of our PRMs are trained for 2 epochs. On smaller datasets (such as
in phase 1 and the first few generations of phase 2) this improves the final
performance over training for just 1 epoch. Additional epochs, up to some point,
don’t noticeably help or hurt performance. On larger datasets, the benefits of
2 epoch training diminishes, but we continue doing it for consistency.
F.2
Scoring
There are multiple ways of using the PRM to score solutions. In general, we
produce a single solution-level score by performing a reduction over step-level
scores, where the step-level score is the probability that the step’s label is pos-
itive. This involves two specific implementation decisions. First, when deter-
mining a step-level score, we either consider a neutral label to be positive or
negative. Second, when determining a solution-level score, we either use the
minimum or the product over step-level scores as a reduction.
We show results from all four scoring strategies in Table 4. The best per-
forming strategy is to take the product of step-level scores and to consider the
neutrals as positives, but the difference in performance between all strategies
is minor. Throughout the rest of this work, we consider neutral steps to be
positive, and we define the solution score to be the product of step-level scores.
Using the product instead of the minimum as the reduction does create a slight
bias against solutions with a larger number of steps.
product
minimum
neutral = positive
78.2%
77.6%
neutral = negative
77.4%
77.8%
Table 4: Best-of-1860 test performance using the PRM with four different scor-
ing strategies.
20


G
Difficulty Breakdown
We show performance of our ORM and PRM on each quintile of the MATH
dataset. We determine quintiles based on the pass rate under the generator.
It is interesting to note that the performance gap is not only apparent on high
difficulty problems: it is in fact apparent across all difficulties. For the lowest
difficulty problems, we see that it is possible to find adversarial examples that
fool the ORM, since the ORM’s performance slightly decreases as the number of
samples increases. In contrast, the PRM remains highly robust over this same
set of samples.
We also see that increasing the number of samples has the largest positive
effect on the highest difficulty problems. This is to be expected, since a large
number of generator samples may be required to find a true and convincing
solution to a hard problem.
101
103
0.0
0.2
0.4
0.6
0.8
1.0
% Problems Solved (Best-of-N)
Quintile 1 (easiest)
PRM
ORM
101
103
Quintile 2
101
103
N = number of solutions per problem
Quintile 3
101
103
Quintile 4
101
103
Quintile 5 (hardest)
Figure 6: A breakdown of ORM vs PRM performance by problem difficulty.
21


H
Synthetic Supervision Details
We can use PRMlarge to provide either outcome or process supervision for
smaller models.
We determine the labels for individual steps based on the
step-level probabilities outputted by PRMlarge. To do this, we set an arbitrary
threshold: any step that PRMlarge assigns a negative label with greater than
20% probability is considered incorrect. We choose this threshold based on the
observation that PRMlarge is slightly miscalibrated in the direction of favoring
positive labels.
To provide process supervision for a solution, we directly return the step-
level labels (positive or negative) provided by PRMlarge, up until the first step
that is marked as negative. This mimics our true human data collection process.
To provide outcome supervision, we mark the solution as correct if and only if
PRMlarge considers every step to be correct (using the same thresholding logic).
22


I
PRM Visualizations
All examples shown come from the large-scale generator (GPT-4).
We note
the pass-rate under the generator to give some sense of the difficulty of these
problems.
I.1
True Positives
These cherry-picked examples show the best-of-1860 solution from the generator
as ranked by the large-scale PRM.
Problem 1. Generator pass-rate: 0.1%. This challenging trigonometry problem
requires applying several identities in a not-at-all obvious succession.
Most
solution attempts fail, because it is hard to choose which identities are actually
helpful. Though successful solutions to this problem are rare, the reward model
correctly recognizes when a valid chain-of-thought has been found.
23


Problem 2. Generator pass-rate: 5.8%. In step 7 and 8, the generator starts
performing guess-and-check. This is a common place the model might hallu-
cinate, by claiming a particular guess is successful when it isn’t. In this case,
the reward model verifies each step and determines that the chain-of-thought is
correct.
Problem 3. Generator pass-rate: 1.7%. The generator successfully applies sev-
eral trigonometric identities to simplify the expression.
24


Problem 4. Generator pass-rate: 4.5%. Here, the generator successfully per-
forms a complex series of polynomial factorizations.
The use of the Sophie-
Germain identity in step 5 is an important step that could be considered in-
sightful.
I.2
True Negatives
Problem 5.
Generator pass-rate: 4.5%.
The generator attempts to use the
difference of squares formula in step 12 on an expression that isn’t in fact a
difference of squares. The reward model catches this mistake.
25


Problem 6. Generator pass-rate: 93.5%. In step 7, the generator makes an
incorrect attempt to simplify an expression.
The reward model catches this
mistake.
Problem 7. Generator pass-rate: 48.0%. In step 11, the generator makes a
simple calculation error. The reward model catches this mistake.
26


Problem 8. Generator pass-rate: 5.8%. The justification in step 8 is strange,
but the reward model lets it slide. In step 9, though, the model incorrectly
factors the expression. The reward model catches this mistake.
I.3
False Positives
Problem 9. Generator pass-rate: 18.5%. The generator makes a subtle counting
error in step 9. On the surface, it appears reasonable to claim that there are 5
ways to exchange the same colored ball since there are 5 colors. However, this
undercounts by a factor of 2, since Bob has 2 choices for which ball to return
to Alice. The reward model is fooled by this mistake.
27


Problem 10. Generator pass-rate: 17.6%. In step 13, the generator attempts
to simplify the equation by combining like terms. It correctly moves and com-
bines the linear terms to the left-hand side, but then mistakenly leaves the
right-hand side untouched. The reward model is fooled by this mistake.
Problem 11. Generator pass-rate: 13.4%. The generator attempts to per-
form long division, but in step 16, it forgets to include the leading zeros in the
repeating part of the decimal. The reward model is fooled by this mistake.
28


Problem 12. Generator pass-rate: 9.1%. In step 4, the generator falsely
claims that the sequence repeats itself every 12 terms, when it’s in fact every
10 terms. This sort of counting mistake occasionally fools the reward model.
29


MATH-SHEPHERD: VERIFY AND REINFORCE LLMS
STEP-BY-STEP WITHOUT HUMAN ANNOTATIONS
Peiyi Wang1†
Lei Li3
Zhihong Shao4
R.X. Xu2
Damai Dai1
Yifei Li5
Deli Chen2
Y. Wu2
Zhifang Sui1
1National Key Laboratory for Multimedia Information Processing, Peking University
2DeepSeek-AI
3The University of Hong Kong
4Tsinghua University
5The Ohio State University
{wangpeiyi9979, nlp.lilei}@gmail.com
li.14042@osu.edu
szf@pku.edu.cn
Project Page: MA T H-SH E P H E R D
ABSTRACT
In this paper, we present an innovative process-oriented math process reward model
called MATH-SHEPHERD, which assigns a reward score to each step of math
problem solutions. The training of MATH-SHEPHERD is achieved using automati-
cally constructed process-wise supervision data, breaking the bottleneck of heavy
reliance on manual annotation in existing work. We explore the effectiveness of
MATH-SHEPHERD in two scenarios: 1) Verification: MATH-SHEPHERD is utilized
for reranking multiple outputs generated by Large Language Models (LLMs);
2) Reinforcement Learning: MATH-SHEPHERD is employed to reinforce LLMs
with step-by-step Proximal Policy Optimization (PPO). With MATH-SHEPHERD, a
series of open-source LLMs demonstrates exceptional performance. For instance,
the step-by-step PPO with MATH-SHEPHERD significantly improves the accuracy
of Mistral-7B (77.9%→84.1% on GSM8K and 28.6%→33.0% on MATH). The
accuracy can be further enhanced to 89.1% and 43.5% on GSM8K and MATH
with the verification of MATH-SHEPHERD, respectively. We believe that automatic
process supervision holds significant potential for the future evolution of LLMs.
LLaMA2-70B
MAmmoTH
LLaMA2-70B
WizardMATH
LLaMA2-70B
MetaMATH*
LLemma-34B
MetaMATH*
DeepSeek-67B
MetaMATH*
GSM8K
70
75
80
85
90
95
Accuracy (%)
72.4
81.6
93.2
90.9
93.3
72.4
81.6
80.4
75.8
82.8
GPT-4 (early): 92.0
GPT-4-0613*: 94.4
Fine-tuned LLMs
+SHEPHERD
LLaMA2-70B
MAmmoTH
LLaMA2-70B
WizardMATH
LLaMA2-70B
MetaMATH*
LLemma-34B
MetaMATH*
DeepSeek-67B
MetaMATH*
MATH
10
20
30
40
50
60
Accuracy (%)
21.1
22.7
45.2
47.3
48.1
21.1
22.7
29.8
34.8
36.8
GPT-4 (early): 42.5
GPT-4-0613*: 56.2
Fine-tuned LLMs
+SHEPHERD
Figure 1: We evaluate the performance of various LLMs with MATH-SHEPHERD on the GSM8K
and MATH datasets. All base models are finetuned with the MetaMath dataset (Yu et al., 2023b).
The +SHEPHERD results are obtained by selecting the best one from 256 candidates using MATH-
SHEPHERD. We observe that MATH-SHEPHERD is compatible with different LLMs. The results of
GPT-4 (early) are from Bubeck et al. (2023).
†Contribution during internship at DeepSeek-AI.
arXiv:2312.08935v3  [cs.AI]  19 Feb 2024


1
INTRODUCTION
Large language models (LLMs) have demonstrated remarkable capabilities across various tasks
(Park et al., 2023; Kaddour et al., 2023; Song et al.; Li et al., 2023a; Wang et al., 2023a; Chen
et al., 2023; Zheng et al., 2023; Wang et al., 2023c), However, even the most advanced LLMs face
challenges in complex multi-step mathematical reasoning problems (Lightman et al., 2023; Huang
et al., 2023). To address this issue, prior research has explored different methodologies, such as pre-
training (Azerbayev et al., 2023), fine-tuning (Luo et al., 2023; Yu et al., 2023b; Wang et al., 2023b),
prompting (Wei et al., 2022; Fu et al., 2022), and verification (Wang et al., 2023d; Li et al., 2023b;
Zhu et al., 2023; Leviathan et al., 2023). Among these techniques, verification has recently emerged
as a favored method. The motivation behind verification is that relying solely on the top-1 result may
not always produce reliable outcomes. A verification model can rerank candidate responses, ensuring
higher accuracy and consistency in the outputs of LLMs. In addition, a good verification model can
also offer invaluable feedback for further improvement of LLMs (Uesato et al., 2022; Wang et al.,
2023b; Pan et al., 2023).
The verification models generally fall into the outcome reward model (ORM) (Cobbe et al., 2021;
Yu et al., 2023a) and process reward model (PRM) (Li et al., 2023b; Uesato et al., 2022; Lightman
et al., 2023; Ma et al., 2023). The ORM assigns a confidence score based on the entire generation
sequence, whereas the PRM evaluates the reasoning path step-by-step. PRM is advantageous due to
several compelling reasons. One major benefit is its ability to offer precise feedback by identifying
the specific location of any errors that may arise, which is a valuable signal in reinforcement learning
and automatic correction. Besides, The PRM exhibits similarities to human behavior when assessing
a reasoning problem. If any steps contain an error, the final result is more likely to be incorrect,
mirroring the way human judgment works. However, gathering data to train a PRM can be an arduous
process. Uesato et al. (2022) and Lightman et al. (2023) utilize human annotators to provide process
supervision annotations, enhancing the performance of PRM. Nevertheless, annotation by humans,
particularly for intricate multi-step reasoning tasks that require advanced annotator skills, can be
quite costly, which hinders the advancement and practical application of PRM.
To tackle the problem, in this paper, we propose an automatic process annotation framework. In-
spired by Monte Carlo Tree Search (Kocsis & Szepesv´
ari, 2006; Coulom, 2006; Silver et al., 2016;
´
Swiechowski et al., 2023), we define the quality of an intermediate step as its potential to deduce
the correct final answer. By leveraging the correctness of the answer, we can automatically gather
step-wise supervision. Specifically, given a math problem with a golden answer and a step-by-step
solution, to achieve the label of a specific step, we utilize a fine-tuned LLM to decode multiple
subsequent reasoning paths from this step. We further validate whether the decoded final answer
matches with the golden answer. If a reasoning step can deduce more correct answers than another, it
would be assigned a higher correctness score.
We use this automatic way to construct the training data for MATH-SHEPHERD, and verify our ideas
on two widely used mathematical benchmarks, GSM8K (Cobbe et al., 2021) and MATH (Hendrycks
et al., 2021). We explore the effectiveness of MATH-SHEPHERD in two scenarios: 1) verification:
MATH-SHEPHERD is utilized for reranking multiple outputs generated by LLMs; 2) reinforcement
learning: MATH-SHEPHERD is employed to reinforce LLMs with step-by-step Proximal Policy
Optimization (PPO). With the verification of MATH-SHEPHERD, a series of open-source LLMs from
7B to 70B demonstrates exceptional performance. For instance, the step-by-step PPO with MATH-
SHEPHERD significantly improves the accuracy of Mistral-7B (77.9%→84.1% on GSM8K and
28.6%→33.0% on MATH). The accuracy can be further enhanced to 89.1% and 43.5% on GSM8K
and MATH with verification. DeepSeek 67B (DeepSeek, 2023) achieves accuracy rates of 93.3% on
the GSM8K dataset and 48.1% on the MATH dataset with verification of MATH-SHEPHERD. To the
best of our knowledge, these results are unprecedented for open-source models that do not rely on
additional tools.
Our main contributions are as follows:
1) We propose a framework to automatically construct process supervision datasets without human
annotations for math reasoning tasks.


2) We evaluate our method on both step-by-step verification and reinforcement learning scenarios.
Extensive experiments on two widely used mathematical benchmarks - GSM8K and MATH, in
addition to a series of LLMs ranging from 7B to 70B, demonstrate the effectiveness of our method.
3) We empirically analyze the key factors for training high-performing process reward models,
shedding light on future directions toward improving reasoning capability with automatic step-by-
step verification and supervision.
2
RELATED WORKS
Improving and eliciting mathematical reasoning abilities of LLMs.
Mathematical reasoning
tasks are one of the most challenging tasks for LLMs. Researchers have proposed various methods to
improve or elicit the mathematical reasoning ability of LLMs, which can be broadly divided into three
groups: 1) pre-training: The pre-training methods (OpenAI, 2023; Anil et al., 2023; Touvron et al.,
2023; Azerbayev et al., 2023) pre-train LLMs on a vast of datasets that are related to math problems,
such as the Proof-Pile and ArXiv (Azerbayev et al., 2023) with a simple next token prediction
objective. 2) fine-tuning: The fine-tuning methods (Yu et al., 2023b; Luo et al., 2023; Yue et al., 2023;
Wang et al., 2023b; Gou et al., 2023) can also enhance the mathematical reasoning ability of LLMs.
The core of fine-tuning usually lies in constructing high-quality question-response pair datasets with
a chain-of-thought reasoning process. and 3) prompting: The prompting methods (Wei et al., 2022;
Zhang et al., 2023; Fu et al., 2022; Bi et al., 2023) aim to elicit the mathematical reasoning ability
of LLMs by designing prompting strategy without updating the model parameters, which is very
convenient and practical.
Mathematical reasoning verification for LLMs.
Except for directly improving and eliciting the
mathematical reasoning potential of LLMs, the reasoning results can be boosted via an extra verifier
for selecting the best answer from multiple decoded candidates. There are two primary types of
verifiers: the Outcome Reward Model (ORM) and the Process Reward Model (PRM). The ORM
allocates a score to the entire solution while the PRM assigns a score to each individual step in
the reasoning process. Recent findings by (Lightman et al., 2023) suggest that PRM outperforms
ORM. In addition to verification, reward models can offer invaluable feedback for further training of
generators (Uesato et al., 2022; Pan et al., 2023). Compared to ORM, PRM provides more detailed
feedback, demonstrating greater potential to enhance generator (Wu et al., 2023). However, training a
PRM requires access to expensive human-annotated datasets (Uesato et al., 2022; Lightman et al.,
2023), which hinders the advancement and practical application of PRM. Therefore, in this paper,
we aim to build a PRM for mathematical reasoning without human annotation, and we explore the
effectiveness of the automatic PRM with both verification and reinforcement learning scenarios.
3
METHODOLOGY
In this section, we first present our task formulation to evaluate the performance of reward models
(§3.1). Subsequently, we outline two typical categories of reward models, ORM and PRM(§3.2).
Then, we introduce our methodology to automatically build the training dataset for PRM(§3.3),
breaking the bottleneck of heavy reliance on manual annotation in existing work (Uesato et al., 2022;
Lightman et al., 2023).
3.1
TASK FORMULATION
We evaluate the performance of the reward model in two scenarios:
Verification
Following (Lightman et al., 2023), we consider a best-of-N selection evaluation
paradigm. Specifically, given a problem p in the testing set, we sample N candidate solutions from a
generator. These candidates are then scored using a reward model, and the highest-scoring solution
is selected as the final answer. An enhanced reward model elevates the likelihood of selecting the
solution containing the correct answer, consequently raising the success rate in solving mathematical
problems for LLMs.


Reinforcement learning
We also use the automatically constructed PRM to supervise LLMs with
step-by-step PPO. In this scenario, we evaluate the accuracy of the LLMs’ greedy decoding output.
An enhanced reward model is instrumental in training higher-performing LLMs.
3.2
REWARD MODELS FOR MATHEMATICAL PROBLEM
ORM
Given a mathematical problem p and its solution s, ORM (P × S →R) assigns a single
real-value to s to indicate whether s is correct. ORM is usually trained with a cross-entropy loss
(Cobbe et al., 2021; Li et al., 2023b):
LORM = ys log rs + (1 −ys) log(1 −rs),
(1)
where ys is the golden answer of the solution s, ys = 1 if s is correct, otherwise ys = 0. rs is
the sigmoid score of s assigned by ORM. The success of the reward model hinges on the effective
construction of the high-quality training dataset. As the math problem usually has a certain answer,
we can automatically construct the training set of ORM by two steps: 1) sampling some candidate
solutions for a problem from a generator; 2) assigning the label to each sampling solution by checking
whether its answer is correct. Although false positives solutions that reach the correct answer with
incorrect reasoning will be misgraded, previous studies have proven that it is still effective for training
a good ORM (Lightman et al., 2023; Yu et al., 2023a).
PRM
Take a step further, PRM (P × S →R+) assigns a score to each reasoning step of s, which
is usually trained with:
LP RM =
K
X
i=1
ysi log rsi + (1 −ysi) log(1 −rsi),
(2)
where ysi is the golden answer of si (the i-th step of s), rsi is the sigmoid score of si assigned by
PRM and K is the number of reasoning steps for s. (Lightman et al., 2023) also conceptualizes the
PRM training as a three-class classification problem, in which each step is classified as either ‘good’,
‘neutral’, or ‘bad’. In this paper, we found that there is not much difference between the binary
and the three classifications, and we regard PRM training as the binary classification. Compared
to ORM, PRM can provide more detailed and reliable feedback (Lightman et al., 2023). However,
there are currently no automated methods available for constructing high-quality PRM training
datasets. Previous works (Uesato et al., 2022; Lightman et al., 2023) typically resort to costly human
annotations. While PRM manages to outperform ORM (Lightman et al., 2023), the annotation cost
invariably impedes both the development and application of PRM.
3.3
AUTOMATIC PROCESS ANNOTATION
In this section, we propose an automatic process annotation framework to mitigate the annotation
cost issues associated with PRM. We first define the quality of a reasoning step, followed by the
introduction of our solution that obviates the necessity for human annotation.
3.3.1
DEFINITION
Inspired by Monto Carlo Tree Search (Kocsis & Szepesv´
ari, 2006; Coulom, 2006; Silver et al., 2016;
´
Swiechowski et al., 2023), we define the quality of a reasoning step as its potential to deduce the
correct answer. This criterion stems from the primary objective of the reasoning process, which
essentially is a cognitive procedure aiding humans or intelligent agents in reaching a well-founded
outcome (Huang & Chang, 2023). Therefore, a step that has the potential to deduce a well-founded
result can be considered a good reasoning step. Analogous to ORM, this definition also introduces
some degree of noise. Nevertheless, we find that it is beneficial for effectively training a good PRM.
3.3.2
SOLUTION
Completion
To quantify and estimate the potential for a give reasoning step si, as shown in
Figure 2, we use a ‘completer’ to finalize N subsequent reasoning processes from this step:
{(si+1,j, · · · , sKj,j, aj)}N
j=1, where aj and Kj are the decoded answer and the total number of
steps for the j-th finalized solution, respectively. Then, we estimate the potential of this step based on
the correctness of all decoded answers A = {aj}N
j=1.


Problem: Let 𝑝𝑝(𝑥𝑥) be a monic polynomial of degree 4.  Three 
of the roots of p(x) are 1, 2, and 3.  Find p(0) + p(4).
Golden Answer: 24
Answer: 20
Solution: 𝑺𝑺= 𝒔𝒔𝟏𝟏, 𝒔𝒔𝟐𝟐, 𝒔𝒔𝟑𝟑, ⋯, 𝒔𝒔𝑲𝑲 
(a) Outcome Annotation: 𝒚𝒚𝑺𝑺= 𝟎𝟎
Problem: …. 
𝒔𝒔𝟏𝟏: Since three of the 
roots of p(x) are 1, 2, and 
3, we can write : p(x) = 
(x - 1)(x - 2)(x - 3)(x - r).
𝒔𝒔𝟐𝟐,𝟏𝟏
𝒔𝒔𝟑𝟑,𝟏𝟏
𝒔𝒔𝑲𝑲𝟏𝟏,𝟏𝟏
⋯
𝒔𝒔𝟐𝟐,𝟐𝟐
𝒔𝒔𝟐𝟐,𝟐𝟐
𝒔𝒔𝑲𝑲𝟐𝟐,𝟐𝟐
⋯
𝒔𝒔𝟐𝟐,𝟑𝟑
𝒔𝒔𝟐𝟐,𝟑𝟑
𝒔𝒔𝑲𝑲𝟑𝟑,𝟑𝟑
⋯
Answer: 24
Answer: 24
Answer: 20
(b): Process Annotation: 𝒚𝒚𝒔𝒔𝟏𝟏
𝑺𝑺𝑺𝑺= 𝟐𝟐
𝟑𝟑 ; 𝒚𝒚𝒔𝒔𝟏𝟏
𝑯𝑯𝑯𝑯=  𝟏𝟏
𝒔𝒔𝒊𝒊,𝒋𝒋: the i-th step of the j-th finalized solution.
𝒔𝒔𝒊𝒊: the i-th step of the solution 𝑺𝑺. 
✓
✓
✗
✗
Figure 2: Comparison for previous automatic outcome annotation and our automatic process an-
notation. (a): automatic outcome annotation assigns a label to the entire solution S, dependent on
the correctness of the answer; (b) automatic process annotation employs a ‘completer’ to finalize N
reasoning processes (N=3 in this figure) for an intermediate step (s1 in this figure), subsequently use
hard estimation (HE) and soft estimation (SE) to annotate this step based on all decoded answers.
Estimation
In this paper, we use two methods to estimate the quality ysi for the step si, hard
estimation (HE) and soft estimation (SE). HE supposes that a reasoning step is good as long as it can
reach the correct answer a∗:
yHE
si
=
1
∃aj ∈A, aj = a∗
0
Otherwise
(3)
SE assumes the quality of a step as the frequency with which it reaches the correct answer:
ySE
si
=
PN
j=1 I(aj = a∗)
N
.
(4)
Once we gather the label of each step, we can train PRM with the cross-entropy loss. In conclusion,
our automatic process annotation framework defines the quality of a step as its potential to deduce
the correct answer and achieve the label of each step by completion and estimation.
3.4
RANKING FOR VERIFICATION
Following (Lightman et al., 2023), we use the minimum score across all steps to represent the final
score of a solution assigned by PRM. We also explore the combination of self-consistency and reward
models following (Li et al., 2023b). In this context, we initially classify solutions into distinct groups
according to their final answers. Following that, we compute the aggregate score for each group.
Formally, the final prediction answer based on N candidate solutions is:
asc+rm = arg max
a
N
X
i=1
I(ai = a) · RM(p, Si).
(5)
Where RM(p, Si) is the score of the i-th solution assigned by ORM or PRM for problem p.
3.5
REINFORCE LEARNING WITH PROCESS SUPERVISION
Upon achieving PRM, we employ reinforcement learning to train LLMs. We implement Proximal
Policy Optimization (PPO) in a step-by-step manner. This method differs from the conventional
strategy that utilizes PPO with ORM, which only offers a reward at the end of the response. Conversely,
our step-by-step PPO offers rewards at the end of each reasoning step.


Models
Verifiers
GSM8K
MATH500
LLaMA2-70B: MetaMATH
Self-Consistency
88.0
39.4
ORM
91.8
40.4
Self-Consistency + ORM
92.0
42.0
MATH-SHEPHERD (Ours)
93.2
44.5
Self-Consistency + MATH-SHEPHERD (Ours)
92.4
45.2
LLemma-34B: MetaMATH
Self-Consistency
82.6
44.2
ORM
90.0
43.7
Self-Consistency + ORM
89.6
45.4
MATH-SHEPHERD (Ours)
90.9
46.0
Self-Consistency + MATH-SHEPHERD (Ours)
89.7
47.3
DeepSeek-67B: MetaMATH
Self-Consistency
88.2
45.4
ORM
92.6
45.3
Self-Consistency + ORM
92.4
47.0
MATH-SHEPHERD (Ours)
93.3
47.0
Self-Consistency + MATH-SHEPHERD (Ours)
92.5
48.1
Table 1: Performances of different LLMs on GSM8K and MATH with different verification strategies.
The reward models are trained based on LLama2-70B and LLemma-34B on GSM8K and MATH,
respectively. The verification is based on 256 outputs.
4
EXPERIMENTS
Datasets
We conduct our experiments using two widely used math reasoning datasets, GSM8K
(Cobbe et al., 2021) and MATH (Hendrycks et al., 2021). For the GSM8K dataset, we leverage the
whole test set in both verification and reinforcement learning scenarios. For the MATH dataset, in the
verification scenario, due to the computation cost, we employ a subset MATH500 that is identical to
the test set of Lightman et al. (2023). The subset consists of 500 representative problems, and we
find that the subset evaluation produces similar results to the full-set evaluation. To assess different
verification methods, we generate 256 candidate solutions for each test problem. We report the mean
accuracy of 3 groups of sampling results. In the reinforcement learning scenario, we use the whole
test set to evaluate the model performance. We train LLMs with MetaMATH (Yu et al., 2023b).
Parameter Setting
Our experiments are based on a series of large language models, LLaMA2-
7B/13B/70B (Touvron et al., 2023), LLemma-7B/34B (Azerbayev et al., 2023), Mistral-7B (Jiang
et al., 2023) and DeepSeek-67B (DeepSeek, 2023). We train the generator and completer for 3 epochs
on MetaMATH. We train the Mistral-7B with a learning rate of 5e-6. For other models, The learning
rates are set to 2e-5, 1e-5, and 6e-6 for the 7B/13B, 34B, and 67B/70B LLMs, respectively. To
construct the training dataset of ORM and PRM, we train 7B and 13B models for a single epoch on
the GSM8K and MATH training sets. Subsequently, we sample 15 solutions per problem from each
model for the training set. Following this, we eliminate duplicate solutions and annotate the solutions
at each step. We use LLemma-7B as the completer with the decoded number N=8. Consequently,
we obtain around 170k solutions for GSM8K and 270k solutions for MATH. For verification, we
choose LLaMA2-70B and LLemma-34B as the base models to train reward models for GSM8K and
MATH, respectively. For reinforcement learning, we choose Mistral-7B as the base model to train
reward models and use it to supervise LLama2-7B and Mistral-7B generators. The reward model is
trained in 1 epoch with a learning rate 1e-6. For the sake of convenience, we train the PRM using
the hard estimation version because it allows us to utilize a standard language modeling pipeline by
selecting two special tokens to represent ‘has potential’ and ‘no potential’ labels, thereby eliminating
the need for any specific model adjustments. In reinforcement learning, the learning rate is 4e-7 and
1e-7 for LLaMA2-7B and Mistral-7B, respectively. The Kullback-Leibler coefficient is set to 0.04.
We implement a cosine learning rate scheduler, employing a minimal learning rate set to 1e-8. We
use 3D parallelism provided by hfai1 to train all models with the max sequence length of 512.
1https://doc.hfai.high-flyer.cn/index.html


Models
GSM8K
MATH
LLaMA2-7B: MetaMATH
66.6
19.2
+ RFT
68.5
19.9
+ ORM-PPO
70.8
20.8
+ MATH-SHEPHERD-step-by-step-PPO (Ours)
73.2
21.6
Mistral-7B: MetaMATH
77.9
28.6
+ RFT
79.0
29.9
+ ORM-PPO
81.8
31.3
+ MATH-SHEPHERD-step-by-step-PPO (Ours)
84.1
33.0
Table 2: Performances of different 7B models on GSM8K and MATH with greedy decoding. We
use the questions in MetaMATH for RFT and PPO training. Both LLaMA2-7B and Mistral-7B are
supervised by Mistral-7B-ORM and -MATH-SHEPHERD.
Baselines and Metrics
In the verification scenario, following (Lightman et al., 2023), we evaluate
the performance of our reward model by comparing it against the Self-consistency (majority voting)
and outcome reward model. The accuracy of the best-of-N solution is utilized as the evaluation
metric. For PRM, the minimum score across all steps is adopted to represent the final score of a
solution. In the reinforcement scenario, we compare our step-by-step supervision with the outcome
supervision provided by ORM, and Rejective Sampling Fine-tuning (RFT) (Yuan et al., 2023), we
sample 8 responses for each question in MetaMATH for RFT. We use the accuracy of LLMs’ greedy
decoding output to assess the performance.
4.1
MAIN RESULTS
MATH-SHEPHERD as verifier
Table 1 presents the performance comparison of various methods
on GSM8K and MATH. We find that: 1) As the verifier, MATH-SHEPHERD consistently outperforms
self-consistency and ORM on two datasets with all generators. Specifically, enhanced by MATH-
SHEPHERD, DeepSeek-67B achieves 93.3% and 48.1% accuracy on GSM8K and MATH; 2) In
comparison to GSM8K, PRM achieves a greater advantage over ORM on the more challenging
MATH dataset; This outcome aligns with the findings in Uesato et al. (2022) and Lightman et al.
(2023). The former discovers that PRM and ORM yield similar results on GSM8K, whereas the latter
shows that PRM significantly outperforms ORM on the MATH dataset. This could be attributed to the
relative simplicity of the GSM8K dataset compared to MATH, i.e., the GSM8K dataset necessitates
fewer steps for problem-solving. As a result, ORM operates efficiently when handling this particular
dataset; 3) In GSM8K, when combined with self-consistency, there’s a drop in performance, whereas
in MATH, performance improves. These results indicate that if the reward model is sufficiently
powerful for a task, combining it with self-consistency may harm the verification performance.
MATH-SHEPHERD as reward model on reinforcement learning
Table 2 presents the performance
of different LLMs with greedy decoding outputs. As is shown: 1) step-by-step PPO significantly
improves the performance of two supervised fine-tuned models. For example, Mistral-7B with
step-by-step PPO achieves 84.1% and 33.0% on the GSM8K and MATH datasets, respectively; 2)
RFT only slightly improves the model performance, we believe this is because MetaMATH already
has conducted some data augmentation strategies like RFT; 3) the vanilla PPO with ORM can also
enhance the model performance. However, it does not perform as well as the step-by-step PPO
supervised by MATH-SHEPHERD, demonstrating the potential of step-by-step supervision.
MATH-SHEPHERD as both reward models and verifiers
We also combine the reinforcement
learning and the verification. As shown in Table 3: 1) reinforcement learning and verification are
complementary. For example, in MATH, step-by-step PPO Mistral-7B outperforms supervised
fine-tuning Mistral-7B 7.2% accuracy with self-consistency as the verifier; The performance gap
is even larger than that of greedy decoding results, i.e., 4.4%; 2) after reinforcement learning, the
vanilla verification methods with only reward models is inferior to self-consistency, we think the


Models
Verifiers
GSM8K
MATH500
Mistral-7B: MetaMATH
Self-Consistency
83.9
35.1
ORM
86.2
36.4
Self-Consistency + ORM
86.6
38.0
MATH-SHEPHERD (Ours)
87.1
37.3
Self-Consistency + MATH-SHEPHERD (Ours)
86.3
38.3
Mistral-7B: MetaMATH
Self-Consistency
87.4
42.3
ORM
87.6
41.3
+step-by-step PPO (Ours)
Self-Consistency + ORM
89.0
43.1
MATH-SHEPHERD (Ours)
88.4
41.1
Self-Consistency + MATH-SHEPHERD (Ours)
89.1
43.5
Table 3: Results of reinforcement learning and verification combination. The reward models are
trained based on Mistral-7B. The verification is based on 256 outputs.
reason is that the initial reward model is not sufficient to supervise the more powerful model after
PPO. These results can also show the potential of iterative reinforcement learning, which we leave
for future work.
5
ANALYSIS
5.1
PERFORMANCE WITH DIFFERENT NUMBER OF CANDIDATE SOLUTIONS
Figure 3 illustrates the performance comparison of various strategies when applied to different
numbers of candidates ranging from 1 to 256 on two benchmarks. The key observations are as
follows: 1) PRM exhibits consistent superior performance when compared to both ORM and majority
voting, with the degree of this superiority becoming more pronounced as N escalates. 2) In MATH, our
automatically annotated datasets outperform the human-annotated PRM800K (Lightman et al., 2023).
We ascribe this superiority to the distribution gap and the data quantity. Specifically, PRM800K is
annotated based on the output from GPT-4, and consequently, a discrepancy arises for the output of
open-source LLaMA models fine-tuned on MetaMATH. Furthermore, when considering the quantity
of data, our automated reward model data exhibits both high scalability and a reduced labeling cost.
Consequently, our dataset is four times larger than that provided in PRM800K. Overall, these results
further underscore the effectiveness and potential of our method.
5.2
QUALITY OF THE AUTOMATIC PROCESS ANNOTATIONS
In this section, we explore the quality of our automatic PRM dataset. To achieve this, we manually
annotate 160 steps sampled from the training set of GSM8K and use different completers to infer
from each step to achieve their label. We find that:
Automatic process annotation exhibits satisfactory quality.
Figure 4(a) demonstrates that uti-
lizing LLaMA2-70B trained on MetaMATH as the completer, the accuracy of the hard estimation
(HE) reaches 86% when N equals 4. This suggests that our automatically constructed dataset is of
high quality. However, we observed a decline in the accuracy of the constructed dataset with further
increases in N. Our analysis indicates that larger values for N may lead to false positives.
Figure 4(b) shows the cross-entropy loss between SE and HE labels compared to the human-annotated
distribution: as N increases, SE progressively aligns closer to the standard distribution, in contrast
to HE which does not exhibit similar behavior. It is essential to note that at N=4, HE achieves an
accuracy of 86%. We can theoretically attain higher quality data exceeding 86% accuracy by utilizing
SE. However, we discovered that the performance of the verifier exhibits no substantial divergence
whether trained with either SE or HE. This may be attributable to the already high-quality annotations
provided by HE.
Furthermore, we also delve into other automatic process annotation methodologies. For instance, (Li
et al., 2023b) employs a natural language inference (NLI) model and a string match rule to annotate a


1
4
16
64
256
N = number of solutions per problem
80.0
82.5
85.0
87.5
90.0
92.5
% Problems Soluved (Best-of-N)
GSM8K
SC
ORM
SHEPHERD
1
4
16
64
256
N = number of solutions per problem
30
35
40
45
% Problems Soluved (Best-of-N)
MATH
SC
ORM
PRM800K
SHEPHERD
Figure 3: Performance of LLaMA2-70B using different verification strategies across different
numbers of solution candidates on GSM8K and MATH.
1
4
16
64
256
N = number of decoded path
80
82
84
86
% Accuracy
7B
13B
70B
1
4
16
64
256
N = number of decoded path
1.0
1.5
2.0
2.5
3.0
Loss
7B:Soft
13B:Soft
70B:Soft
70B:Hard
1
4
16
64
256
N = number of decoded path
1
2
3
4
Loss
Normal
Weak
Augmented
Figure 4: Quality of process annotation on GSM8K. (a): Accuracy of the process annotation using
different completer; (b): Loss of the process annotation using different completer; (c): Loss of the
process annotation using the same completer with different training data.
given step. The NLI-based method annotates a step as correct if it is entailment with any step in the
reference solutions. The Rule-based method annotates a step as correct if its support number precisely
matches that of any steps in the reference solutions. As demonstrated in Table 4, our annotation
strategy exhibits substantial superiority over the two approaches.
The ability of the LLM completer plays an important role in the data quality.
We employ
a completer to finalize multiple subsequent reasoning processes for a given step. Therefore, we
investigate the impact of the LLM completer.
Figure 4(b) presents the cross-entropy loss across diverse completers trained on MetaMath. The
results indicate that a larger completer is adept at generating superior-quality datasets. Figure 4(c)
depicts the cross-entropy loss of LLaMA2-70B trained with different datasets. ‘Normal’ denotes
the original GSM8K training dataset; ‘Weak’ refers to the Normal set excluding examples whose
questions are in our 160 evaluation set; while ‘Augmented’ symbolizes MetaMath, an augmented
version of the Normal set.
The findings suggest that high-quality training sets allow the model to operate more proficiently as a
completer. Importantly, the ‘Weak’ set exhibits a markedly larger loss than other datasets. This insight
drives us to infer that LLMs should acquire the questions in advance to enhance their performance
as completers. We can also conjecture that a stronger foundational model, coupled with superior
training data, could further enhance the quality of automatic annotation.
5.3
INFLUENCE OF THE PRE-TRAINED BASE MODELS
To conduct an exhaustive evaluation of MATH-SHEPHERD’s effectiveness, we performed a diverse
range of experiments using model sizes 7B, 13B, and 70B.


Methods
Models
Accuracy (%)
Loss
DIVERSE-NLI (Li et al., 2023b)
DeBERTa (He et al., 2020)
61.3
5.43
DIVERSE-NLI (Li et al., 2023b)
LLaMA2-13B
75.6
3.27
DIVERSE-Rule (Li et al., 2023b)
-
75.0
3.43
MATH-SHEPHERD
LLaMA2-13B (N = 4)
85.0
2.05
Table 4: The comparison between NLI/Rule-based automatic process annotation methods from Li
et al. (2023b) and our method.
1
4
16
64
256
(a) N = number of solutions per problem
62
64
66
68
70
72
74
% Problems Soluved (Best-of-N)
Generator:7B; Verifier:7B
SC
ORM
SHEPHERD
1
4
16
64
256
(b) N = number of solutions per problem
68
70
72
74
76
78
80
Generator:13B; Verifier:13B
SC
ORM
SHEPHERD
1
4
16
64
256
(c) N = number of solutions per problem
81
82
83
84
85
86
87
88
Generator:70B; Verifier:7B
SC
ORM
SHEPHERD
1
4
16
64
256
(d) N = number of solutions per problem
65
70
75
80
85
Generator:7B; Verifier:70B
SC
ORM
SHEPHERD
Figure 5: Performance of different verification strategies on different sizes of generators and verifiers.
Figures 5(a), 5(b), and 3(a) display the results from the 7B, 13B, and 70B generators paired with
equal-sized reward models, respectively. It becomes evident that PRM exhibits superiority over
self-consistency and ORM across all sizes of base models. Moreover, bigger reward models prove
to be more robust; for instance, the accuracy of the 70B reward models escalates as the number of
candidate solutions rises, while the 7B reward models show a decreasing trend.
Figure 5(c) and 5(d) presents the performance of 7B and 70B generators interfaced with different-
sized reward models. The findings illustrate that utilizing a larger reward model to validate the output
of a smaller generator significantly enhances performance. Conversely, when a smaller reward model
is employed to validate the output of a larger generator, the verification process adversely impacts the
model’s performance compared to SC. These results substantiate that we should utilize a more potent
reward model for validating or supervising the generator.
5.4
INFLUENCE OF THE NUMBER OF DATA
We delve deeper into the analysis of PRM and ORM by utilizing varying quantities of training data.
As depicted in Figure 6(a), it is clear that PRM exhibits superior data efficiency. Specifically, it
outperforms ORM by approximately 4% accuracy when applying a modestly sized training dataset
(i.e., 10k instances). Furthermore, PRM seems to have a higher potential ceiling than ORM. These
observations highlight the efficacy of PRM for verification purposes.
5.5
OUT-OF-DISTRIBUTION PERFORMANCE
To further demonstrate the effectiveness of our method, we conduct an out-of-distribution evaluation
on the Hungarian national final exam2, which consists of 33 questions. The total score of these
questions is 100. We use the LLemma-34B trained on MetaMATH to serve as the generator and
generate 256 candidate solutions for each question. We use LLemma-34B-ORM and LLemma-
34B-PRM to select the solution for each question. As shown in Figure 6(b): 1) both LLemma-
34B-ORM and LLemma-34B-PRM outperform the origin LLemma-34B, showing the reward model
can generalize to other domains; 2) PRM outperforms ORM 9 scores, further demonstrating the
superiority of PRM.
2https://huggingface.co/datasets/keirp/hungarian_national_hs_finals_
exam


10k
20k
40k
80k
160k
Number of training solutions
88
90
92
% Problems Soluved (Best-of-256)
SC
ORM
SHEPHERD
Greedy
ORM
SHEPHERD
Methods
30
40
50
60
70
Score
46.0
54.0
63.0
Figure 6: (a): Performance of different reward models using different numbers of training data; (b)
performance of different verification strategies on the out-of-distribution Hungarian national exam.
Question
The first term of an arithmetic sequence is 18. The sum of the first six terms of the sequence is equal to the sum of the first
seven terms. Show that the sum of the first 13 terms of the sequence is 0 and calculate the 13th term of the sequence.
MATH-SHEPHERD
Step 1: Let the common difference of the arithmetic sequence be d. MATH-SHEPHERD: 0.99.
Step 2: The sum of the first six terms of the sequence is 18 + (18 + d) + (18 + 2d) + (18 + 3d) + (18 + 4d) +
(18 + 5d) = 108 + 30d. MATH-SHEPHERD: 0.85.
......
Step 6: Therefore, d = −3. MATH-SHEPHERD: 0.85.
......
Step 8: The 13th term of the sequence is 18 −36 = −18. MATH-SHEPHERD: 0.93. ORM Score: 0.73.
ORM
Step 1: Let the common difference of the arithmetic sequence be d. MATH-SHEPHERD: 0.99.
Step 2: The sum of the first six terms is 18 + (18 + d) + (18 + 2d) + (18 + 3d) + (18 + 4d) + (18 + 5d) =
90 + 15d. MATH-SHEPHERD: 0.53.
......
Step 6: Dividing by −6, we find that d = −2. MATH-SHEPHERD: 0.38.
......
Step 9: The 13th term of the sequence is 18 −26 = −8. MATH-SHEPHERD: 0.38. ORM Score: 0.84.
Table 5: A case study from the Hungarian national exam. Red text denotes the mistake that ORM
fails to detect.
We also conduct a case study to intuitively demonstrate the effectiveness of MATH-SHEPHERD. As
outlined in Table 5, when presented with a question from the Hungarian national final exam, our
MATH-SHEPHERD accurately selected the correct solution from a pool of 256 potential solutions,
which ORM failed. Moreover, MATH-SHEPHERD displayed superior discernment by precisely
identifying incorrect steps within the solutions selected by ORM. Notably, it recognized errors in
Step 2, Step 6, and Step 9 and so on, and subsequently assigned them lower scores relative to those
for steps present in the correct solutions.
6
LIMITATIONS
Our paper has some limitations, which we leave for future work:
The computational cost of the completion process.
To determine the label of each reasoning
step, we utilize a ‘completer’ to decode N subsequent reasoning processes. We observe that as N
increases, so does the quality of automatic annotations. However, this completion process demands a
lot of computing resources, potentially imposing a limitation on the usage of our method. Despite
this limitation, the cost remains significantly lower than human annotation. Furthermore, we are
optimistic that advancements in efficient inference techniques such as speculative decoding (Xia
et al., 2022; Leviathan et al., 2023) and vLLM (Kwon et al., 2023) could mitigate this limitation.
The automatic process annotation consists of noise.
Similar to the automatic outcome annotation,
our automatic process annotation also has noise. Despite this, our experiments verify the efficacy
of our method for training a PRM. In particular, the PRM trained on our dataset outperforms the


human-annotated PRM800K dataset. However, a noticeable gap remains between PRM800K and the
candidate responses generated by the open-source models utilized in this study, which may result in
the invalidation of PRM800K. As a result, the impact of this potential noise on PRM performance
is still undetermined. A comprehensive comparison between human and automated annotations is
envisaged for future studies. Furthermore, we assert that integrating human and automated process
annotations could play a vital role in constructing robust and efficient process supervision.
7
CONCLUSION
In this paper, we introduce a process-oriented math verifier called MATH-SHEPHERD, which assigns a
reward score to each step of the LLM’s outputs on math problems. The training of MATH-SHEPHERD
is achieved using automatically constructed process-wise supervision data, thereby eradicating the
necessity for labor-intensive human annotation. Remarkably, this automatic methodology correlates
strongly with human annotations. Extensive experiments in both verification and reinforcement
learning scenarios demonstrate the effectiveness of our method.
REFERENCES
Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos,
Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm 2 technical report. arXiv
preprint arXiv:2305.10403, 2023.
Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen McAleer, Albert Q
Jiang, Jia Deng, Stella Biderman, and Sean Welleck. Llemma: An open language model for
mathematics. arXiv preprint arXiv:2310.10631, 2023.
Zhen Bi, Ningyu Zhang, Yinuo Jiang, Shumin Deng, Guozhou Zheng, and Huajun Chen. When do
program-of-thoughts work for reasoning? arXiv preprint arXiv:2308.15452, 2023.
S´
ebastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar,
Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence:
Early experiments with gpt-4. arXiv preprint arXiv:2303.12712, 2023.
Liang Chen, Yichi Zhang, Shuhuai Ren, Haozhe Zhao, Zefan Cai, Yuchi Wang, Peiyi Wang, Tianyu
Liu, and Baobao Chang. Towards end-to-end embodied decision making via multi-modal large
language model: Explorations with gpt4-vision and beyond. arXiv preprint arXiv:2310.02071,
2023.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser,
Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve
math word problems. arXiv preprint arXiv:2110.14168, 2021.
R´
emi Coulom. Efficient selectivity and backup operators in monte-carlo tree search. In International
conference on computers and games, pp. 72–83. Springer, 2006.
DeepSeek. Deepseek llm: Let there be answers. https://github.com/deepseek-ai/
DeepSeek-LLM, 2023.
Yao Fu, Hao Peng, Ashish Sabharwal, Peter Clark, and Tushar Khot. Complexity-based prompting
for multi-step reasoning. arXiv preprint arXiv:2210.00720, 2022.
Zhibin Gou, Zhihong Shao, Yeyun Gong, Yujiu Yang, Minlie Huang, Nan Duan, Weizhu Chen,
et al. Tora: A tool-integrated reasoning agent for mathematical problem solving. arXiv preprint
arXiv:2309.17452, 2023.
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: Decoding-enhanced bert
with disentangled attention. arXiv preprint arXiv:2006.03654, 2020.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song,
and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv
preprint arXiv:2103.03874, 2021.


Jie Huang and Kevin Chen-Chuan Chang. Towards reasoning in large language models: A sur-
vey. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Findings of the As-
sociation for Computational Linguistics: ACL 2023, pp. 1049–1065, Toronto, Canada, July
2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.findings-acl.67. URL
https://aclanthology.org/2023.findings-acl.67.
Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song,
and Denny Zhou. Large language models cannot self-correct reasoning yet. arXiv preprint
arXiv:2310.01798, 2023.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot,
Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al.
Mistral 7b. arXiv preprint arXiv:2310.06825, 2023.
Jean Kaddour, Joshua Harris, Maximilian Mozes, Herbie Bradley, Roberta Raileanu, and Robert
McHardy. Challenges and applications of large language models. arXiv preprint arXiv:2307.10169,
2023.
Levente Kocsis and Csaba Szepesv´
ari. Bandit based monte-carlo planning. In European conference
on machine learning, pp. 282–293. Springer, 2006.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph
Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model
serving with pagedattention.
In Proceedings of the 29th Symposium on Operating Systems
Principles, pp. 611–626, 2023.
Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative
decoding. In International Conference on Machine Learning, pp. 19274–19286. PMLR, 2023.
Lei Li, Yuwei Yin, Shicheng Li, Liang Chen, Peiyi Wang, Shuhuai Ren, Mukai Li, Yazheng Yang,
Jingjing Xu, Xu Sun, et al. M3it: A large-scale dataset towards multi-modal multilingual instruction
tuning. arXiv preprint arXiv:2306.04387, 2023a.
Yifei Li, Zeqi Lin, Shizhuo Zhang, Qiang Fu, Bei Chen, Jian-Guang Lou, and Weizhu Chen.
Making language models better reasoners with step-aware verifier. In Anna Rogers, Jordan Boyd-
Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association
for Computational Linguistics (Volume 1: Long Papers), pp. 5315–5333, Toronto, Canada, July
2023b. Association for Computational Linguistics. doi: 10.18653/v1/2023.acl-long.291. URL
https://aclanthology.org/2023.acl-long.291.
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan
Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. arXiv preprint
arXiv:2305.20050, 2023.
Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng,
Qingwei Lin, Shifeng Chen, and Dongmei Zhang. Wizardmath: Empowering mathematical
reasoning for large language models via reinforced evol-instruct. arXiv preprint arXiv:2308.09583,
2023.
Qianli Ma, Haotian Zhou, Tingkai Liu, Jianbo Yuan, Pengfei Liu, Yang You, and Hongxia Yang.
Let’s reward step by step: Step-level reward model as the navigators for reasoning. arXiv preprint
arXiv:2310.10080, 2023.
OpenAI. GPT-4 technical report. CoRR, abs/2303.08774, 2023. doi: 10.48550/arXiv.2303.08774.
URL https://doi.org/10.48550/arXiv.2303.08774.
Sarah Pan, Vladislav Lialin, Sherin Muckatira, and Anna Rumshisky. Let’s reinforce step by step.
arXiv preprint arXiv:2311.05821, 2023.
Joon Sung Park, Joseph O’Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S
Bernstein. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th
Annual ACM Symposium on User Interface Software and Technology, pp. 1–22, 2023.


David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche,
Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering
the game of go with deep neural networks and tree search. nature, 529(7587):484–489, 2016.
Yifan Song, Weimin Xiong, Dawei Zhu, Cheng Li, Ke Wang, Ye Tian, and Sujian Li. Restgpt: Con-
necting large language models with real-world applications via restful apis. corr, abs/2306.06624,
2023. doi: 10.48550. arXiv preprint arXiv.2306.06624.
Maciej ´
Swiechowski, Konrad Godlewski, Bartosz Sawicki, and Jacek Ma´
ndziuk. Monte carlo tree
search: A review of recent modifications and applications. Artificial Intelligence Review, 56(3):
2497–2562, 2023.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay
Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation
and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023.
Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia
Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process-and
outcome-based feedback. arXiv preprint arXiv:2211.14275, 2022.
Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and
Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv
preprint arXiv:2305.16291, 2023a.
Peiyi Wang, Lei Li, Liang Chen, Feifan Song, Binghuai Lin, Yunbo Cao, Tianyu Liu, and Zhifang Sui.
Making large language models better reasoners with alignment. arXiv preprint arXiv:2309.02144,
2023b.
Peiyi Wang, Lei Li, Liang Chen, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and
Zhifang Sui. Large language models are not fair evaluators. arXiv preprint arXiv:2305.17926,
2023c.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha
Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language
models. In The Eleventh International Conference on Learning Representations, ICLR 2023,
Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023d. URL https://openreview.net/
pdf?id=1PL1NIMMrw.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V.
Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In
NeurIPS, 2022. URL http://papers.nips.cc/paper_files/paper/2022/hash/
9d5609613524ecf4f15af0f7b31abca4-Abstract-Conference.html.
Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A Smith,
Mari Ostendorf, and Hannaneh Hajishirzi. Fine-grained human feedback gives better rewards for
language model training. arXiv preprint arXiv:2306.01693, 2023.
Heming Xia, Tao Ge, Furu Wei, and Zhifang Sui. Lossless speedup of autoregressive translation with
generalized aggressive decoding. arXiv preprint arXiv:2203.16487, 2022.
Fei Yu, Anningzhe Gao, and Benyou Wang. Outcome-supervised verifiers for planning in mathemati-
cal reasoning. arXiv preprint arXiv:2311.09724, 2023a.
Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo
Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for
large language models. arXiv preprint arXiv:2309.12284, 2023b.
Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Chuanqi Tan, and Chang Zhou. Scaling
relationship on learning mathematical reasoning with large language models. arXiv preprint
arXiv:2308.01825, 2023.
Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen.
Mammoth: Building math generalist models through hybrid instruction tuning. arXiv preprint
arXiv:2309.05653, 2023.


Yifan Zhang, Jingqin Yang, Yang Yuan, and Andrew Chi-Chih Yao. Cumulative reasoning with large
language models. arXiv preprint arXiv:2308.04371, 2023.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang,
Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and
chatbot arena. arXiv preprint arXiv:2306.05685, 2023.
Xinyu Zhu, Junjie Wang, Lin Zhang, Yuxiang Zhang, Yongfeng Huang, Ruyi Gan, Jiaxing Zhang,
and Yujiu Yang. Solving math word problems via cooperative reasoning induced language models.
In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual
Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 4471–4485,
Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.
acl-long.245. URL https://aclanthology.org/2023.acl-long.245.