# LongBench v2 / 66f95126bb02136c067c5070

task_id: 557c57e4-7d0b-54ed-9e52-736b2b460ce3
task_key: train--66f95126bb02136c067c5070
task_revision_id: 2

{"choice_A":"math-shepherd uses reinforcement learning to improve model capabilities and provides specific training methods.","choice_B":"math-shepherd proposes a method for automatically labeling PRM, which simplifies the manual labeling part in the lets verify article.","choice_C":"math-shepherd is compared with the Self-consistency method, while the lets verify article does not compare.","choice_D":"math-shepherd introduces two methods of estimating rewards, hard and soft.","context":"Let’s Verify Step by Step\nOpenAI\nAbstract\nIn recent years, large language models have greatly improved in their\nability to perform complex multi-step reasoning.\nHowever, even state-\nof-the-art models still regularly produce logical mistakes. To train more\nreliable models, we can turn either to outcome supervision, which provides\nfeedback for a final result, or process supervision, which provides feedback\nfor each intermediate reasoning step. Given the importance of training\nreliable models, and given the high cost of human feedback, it is impor-\ntant to carefully compare the both methods. Recent work has already\nbegun this comparison, but many questions still remain. We conduct our\nown investigation, finding that process supervision significantly outper-\nforms outcome supervision for training models to solve problems from the\nchallenging MATH dataset. Our process-supervised model solves 78% of\nproblems from a representative subset of the MATH test set. Additionally,\nwe show that active learning significantly improves the efficacy of process\nsupervision. To support related research, we also release PRM800K, the\ncomplete dataset of 800,000 step-level human feedback labels used to train\nour best reward model.\n1\nIntroduction\nLarge language models are capable of solving tasks that require complex multi-\nstep reasoning by generating solutions in a step-by-step chain-of-thought format\n(Nye et al., 2021; Wei et al., 2022; Kojima et al., 2022). However, even state-\nof-the-art models are prone to producing falsehoods — they exhibit a tendency\nto invent facts in moments of uncertainty (Bubeck et al., 2023). These hallu-\ncinations (Maynez et al., 2020) are particularly problematic in domains that\nrequire multi-step reasoning, since a single logical error is enough to derail a\nmuch larger solution. Detecting and mitigating hallucinations is essential to\nimprove reasoning capabilities.\n∗Primary authors. Correspondence to: Karl Cobbe <karl@openai.com>\n1\narXiv:2305.20050v1  [cs.LG]  31 May 2023\n\n\nOne effective method involves training reward models to discriminate be-\ntween desirable and undesirable outputs. The reward model can then be used\nin a reinforcement learning pipeline (Ziegler et al., 2019; Stiennon et al., 2020;\nNakano et al., 2021; Ouyang et al., 2022) or to perform search via rejection sam-\npling (Nichols et al., 2020; Shen et al., 2021; Cobbe et al., 2021). While these\ntechniques are useful, the resulting system is only as reliable as the reward\nmodel itself. It is therefore important that we study how to most effectively\ntrain reliable reward models.\nIn closely related work, Uesato et al. (2022) describe two distinct meth-\nods for training reward models: outcome supervision and process supervision.\nOutcome-supervised reward models (ORMs) are trained using only the final\nresult of the model’s chain-of-thought, while process-supervised reward models\n(PRMs) receive feedback for each step in the chain-of-thought. There are com-\npelling reasons to favor process supervision. It provides more precise feedback,\nsince it specifies the exact location of any errors that occur. It also has sev-\neral advantages relevant to AI alignment: it is easier for humans to interpret,\nand it more directly rewards models for following a human-endorsed chain-of-\nthought. Within the domain of logical reasoning, models trained with outcome\nsupervision regularly use incorrect reasoning to reach the correct final answer\n(Zelikman et al., 2022; Creswell et al., 2022).\nProcess supervision has been\nshown to mitigate this misaligned behavior (Uesato et al., 2022).\nDespite these advantages, Uesato et al. (2022) found that outcome supervi-\nsion and process supervision led to similar final performance in the domain of\ngrade school math. We conduct our own detailed comparison of outcome and\nprocess supervision, with three main differences: we use a more capable base\nmodel, we use significantly more human feedback, and we train and test on the\nmore challenging MATH dataset (Hendrycks et al., 2021).\nOur main contributions are as follows:\n1. We show that process supervision can train much more reliable reward\nmodels than outcome supervision. We use our state-of-the-art PRM to\nsolve 78.2% of problems from a representative subset of the MATH test\nset.\n2. We show that a large reward model can reliably approximate human su-\npervision for smaller reward models, and that it can be used to efficiently\nconduct large-scale data collection ablations.\n3. We show that active learning leads to a 2.6× improvement in the data\nefficiency of process supervision.\n4. We release our full process supervision dataset, PRM800K, to promote\nrelated research.\n2\n\n\n2\nMethods\nWe perform a comparison of outcome and process supervision, following a sim-\nilar methodology to Uesato et al. (2022). Outcome supervision can be provided\nwithout humans, since all problems in the MATH dataset have automatically\ncheckable answers. In contrast, there is no simple way to automate process su-\npervision. We therefore rely on human data-labelers to provide process super-\nvision, specifically by labelling the correctness of each step in model-generated\nsolutions.\nWe conduct experiments in two separate regimes: large-scale and small-\nscale. Each has its own advantages, and they offer complimentary perspectives.\nAt large-scale, we finetune all models from GPT-4 (OpenAI, 2023). We focus\non advancing the state-of-the-art by training the most reliable ORM and PRM\npossible. Unfortunately the training sets for these reward models are not directly\ncomparable, for reasons we will discuss in Section 3. These models are therefore\nnot ideal for making an apples-to-apples comparison of outcome and process\nsupervision. To address this flaw, we also train models at small-scale, where\nwe can conduct a more direct comparison. In order to remove our dependence\non costly human feedback, we use a large-scale model to supervise small-scale\nmodel training. This setup enables us to conduct several important ablations\nthat would otherwise be infeasible.\n2.1\nScope\nAt each model scale, we use a single fixed model to generate all solutions. We\ncall this model the generator. We do not attempt to improve the generator with\nreinforcement learning (RL). When we discuss outcome and process supervision,\nwe are specifically referring to the supervision given to the reward model. We do\nnot discuss any supervision the generator would receive from the reward model\nif trained with RL. Although finetuning the generator with RL is a natural next\nstep, it is intentionally not the focus of this work.\nWe instead focus exclusively on how to train the most reliable reward model\npossible. We evaluate a reward model by its ability to perform best-of-N search\nover uniformly sampled solutions from the generator. For each test problem we\nselect the solution ranked highest by the reward model, automatically grade it\nbased on its final answer, and report the fraction that are correct. A reward\nmodel that is more reliable will select the correct solution more often.\n2.2\nBase Models\nAll large-scale models are finetuned from the base GPT-4 model (OpenAI, 2023).\nThis model has been pretrained solely to predict the next token; it has not been\npretrained with any Reinforcement Learning from Human Feedback (RLHF)\n(Christiano et al., 2017). The small-scale base models are similar in design to\nGPT-4, but they were pretrained with roughly 200 times less compute. As an\nadditional pretraining step, we finetune all models on a dataset of roughly 1.5B\n3\n\n\nFigure 1: A screenshot of the interface used to collect feedback for each step in\na solution.\nmath-relevant tokens, which we call MathMix.\nSimilar to Lewkowycz et al.\n(2022), we find that this improves the model’s mathematical reasoning capabil-\nities. Details on how this dataset was constructed can be found in Appendix A.\n2.3\nGenerator\nTo make parsing individual steps easier, we train the generator to produce\nsolutions in a newline delimited step-by-step format. Specifically, we few-shot\ngenerate solutions to MATH training problems, filter to those that reach the\ncorrect final answer, and finetune the base model on this dataset for a single\nepoch. This step is not intended to teach the generator new skills; it is intended\nonly to teach the generator to produce solutions in the desired format.\n2.4\nData Collection\nTo collect process supervision data, we present human data-labelers with step-\nby-step solutions to MATH problems sampled by the large-scale generator.\nTheir task is to assign each step in the solution a label of positive, negative,\nor neutral, as shown in Figure 1. A positive label indicates that the step is cor-\nrect and reasonable. A negative label indicates that the step is either incorrect\nor unreasonable. A neutral label indicates ambiguity. In practice, a step may\nbe labelled neutral if it is subtly misleading, or if it is a poor suggestion that\nis technically still valid. We permit neutral labels since this allows us to defer\nthe decision about how to handle ambiguity: at test time, we can treat neutral\nlabels as either positive or negative. A more detailed description of the labelling\ninstructions is provided in Appendix D.\nWe label solutions exclusively from the large-scale generator in order to\nmaximize the value of our limited human-data resource. We refer to the en-\ntire dataset of step-level labels collected as PRM800K. The PRM800K training\nset contains 800K step-level labels across 75K solutions to 12K problems. To\n4\n\n\nminimize overfitting, we include data from 4.5K MATH test problems in the\nPRM800K training set, and we therefore evaluate our models only on the re-\nmaining 500 MATH test problems. More details about this test set can be found\nin Appendix C.\nDuring data collection, we must decide which solutions to surface to data-\nlabelers. The most straightforward strategy is to uniformly surface solutions\nproduced by the generator. However, if we surface solutions that make obvious\nerrors, the human feedback we get is less valuable. We would prefer to surface\nsolutions that are more likely to fool our best reward model. To that end, we at-\ntempt to strategically select which solutions to show data-labelers. Specifically,\nwe choose to surface convincing wrong-answer solutions. We use the term con-\nvincing to refer to solutions that are rated highly by our current best PRM, and\nwe use wrong-answer to refer to solutions that reach an incorrect final answer.\nWe use this slightly verbose phrasing to emphasize the fact that correctness is\ndetermined solely by checking the final answer, a process which occasionally\nleads to misgraded solutions. We expect to gain more information from labeling\nconvincing wrong-answer solutions, since we know the PRM is mistaken about\nat least one step in each such solution.\nIn addition to using this selection strategy, we also iteratively re-train our\nPRM using the latest data at several points in the data collection process. At\neach iteration, we generate N solutions per problem and surface only the top K\nmost convincing wrong-answer solutions to data-labelers. We experiment with\neither applying this top-K filtering at a problem level (K solutions per problem)\nor globally across the dataset (K solutions in total, unequally distributed among\nproblems). Since the data collection process is expensive, it was not feasible\nto conduct at-scale ablations of these decisions. However, we perform several\nsurrogate ablations in Section 4, using our largest PRM as a labelling oracle for\na smaller PRM. More details about data collection can be found in Appendix B.\n2.5\nOutcome-supervised Reward Models (ORMs)\nWe train ORMs following a similar methodology to Cobbe et al. (2021). We\nuniformly sample a fixed number of solutions per problem from the generator,\nand we train the ORM to predict whether each solution is correct or incorrect.\nIn practice, we usually determine correctness by automatically checking the\nfinal answer, but in principle these labels could be provided by humans. At test\ntime, we use the ORM’s prediction at the final token as the overall score for the\nsolution. We note the automatic grading used to determine ORM targets is not\nperfectly reliable: false positives solutions that reach the correct answer with\nincorrect reasoning will be misgraded.\nWe discuss additional ORM training\ndetails in Appendix E.\n2.6\nProcess-supervised Reward Models (PRMs)\nWe train PRMs to predict the correctness of each step after the last token in\neach step. This prediction takes the form of a single token, and we maximize the\n5\n\n\nFigure 2: Two solutions to the same problem, graded by the PRM. The solution\non the left is correct while the solution on the right is incorrect.\nA green\nbackground indicates a high PRM score, and a red background indicates a low\nscore. The PRM correctly identifies the mistake in the incorrect solution.\nlog-likelihood of these target tokens during training. The PRM can therefore\nbe trained in a standard language model pipeline without any special accom-\nmodations. To determine the step-level predictions at test time, it suffices to\nperform a single PRM forward pass over the whole solution. We visualize large-\nscale PRM scores for two different solutions in Figure 2. To compare multiple\nsolutions, it is necessary to compute a single score for each solution. This is an\nimportant but straightforward detail: we define the PRM score for a solution to\nbe the probability that every step is correct under the PRM. We implement this\nas the product of the correctness probabilities for each step. We describe other\npossible scoring strategies and additional PRM training details in Appendix F.\nWhen we provide process supervision, we deliberately choose to supervise\nonly up to the first incorrect step. This makes the comparison between out-\ncome and process supervision more straightforward. For correct solutions, both\nmethods provide the same information, namely that every step is correct. For\nincorrect solutions, both methods reveal the existence of at least one mistake,\nand process supervision additionally reveals the precise location of that mistake.\nIf we were to provide additional process supervision beyond the first mistake,\nthen process supervision would have an even greater information advantage.\nThis decision also keeps the labelling cost similar for humans: without relying\non an easy-to-check final answer, determining the correctness of a solution is\nequivalent to identifying its first mistake. While most MATH problems do have\neasy-to-check final answers, we expect this to not remain true in more complex\ndomains.\n6\n\n\nORM\nPRM\nMajority Voting\n% Solved (Best-of-1860)\n72.4\n78.2\n69.6\n101\n102\n103\nN = number of solutions per problem\n62\n64\n66\n68\n70\n72\n74\n76\n78\n% Problems Solved (Best-of-N)\nProcess-Supervised RM\nOutcome-Supervised RM\nMajority Voting\nFigure 3: A comparison of outcome-supervised and process-supervised reward\nmodels, evaluated by their ability to search over many test solutions. Majority\nvoting is shown as a strong baseline. For N ≤1000, we visualize the variance\nacross many subsamples of the 1860 solutions we generated in total per problem.\n3\nLarge-scale Supervision\nWe train the large-scale PRM using the step-level labels in PRM800K. To ensure\nthe large-scale ORM baseline is as strong as possible, we train on 100 uniform\nsamples per problem from the generator. This means the ORM training set has\nno overlap with PRM800K, and it is an order of magnitude larger. Although\nthese two training sets are not directly comparable, each represents our best\nattempt to advance the state-of-the-art with each form of supervision. We note\nthat training the ORM solely on PRM800K solutions would be problematic,\nsince our active learning strategy has heavily biased the dataset towards wrong-\nanswer solutions. We did explore training the ORM on a superset of PRM800K\nsolutions, by mixing in uniformly sampled solutions, but we found that this did\nnot improve ORM performance.\nFigure 3 shows how the best-of-N performance of each reward model varies\nas a function of N. Since majority voting is known to be a strong baseline (Wang\net al., 2022; Lewkowycz et al., 2022), we also include this method as a point of\ncomparison. While the ORM performs slightly better than the majority voting\nbaseline, the PRM strongly outperforms both. Not only does the PRM reach\nhigher performance for all values of N, but the performance gap widens as N\nincreases. This indicates that the PRM is more effective than both the ORM and\nmajority voting at searching over a large number of model-generated solutions.\n7\n\n\n100\n101\n102\nNumber of solutions labelled per problem\n25\n30\n35\n40\n45\n50\n55\n60\n% Problems Solved (Best-of-500)\nPRM + Active Learning\nPRM (PRMlarge supervised)\nORM (PRMlarge supervised)\nORM (final-answer supervised)\n(a)\nFour\nseries\nof\nreward\nmodels\ntrained using different data collection\nstrategies, compared across training\nsets of varying sizes.\n100\n101\n102\n103\nN = number of solutions per problem\n20\n25\n30\n35\n40\n45\n50\n55\n60\n% Problems Solved (Best-of-N)\nPRM (PRMlarge supervised)\nORM (PRMlarge supervised)\nORM (final-answer supervised)\n(b) Three reward models trained on\n200 samples/problem using different\nforms of supervision, compared across\nmany test-time compute budgets.\nFigure 4: A comparison of different forms of outcome and process supervision.\nMean and standard deviation is shown across three seeds.\nWe experimented with using RM-weighted voting (Li et al., 2022; Uesato et al.,\n2022) to combine the benefits of the PRM and majority voting, but this did not\nnoticeably improve performance. We use a specific subset of the MATH test set\nfor evaluation, which we describe in Appendix C. We further break down these\nresults by problem difficulty in Appendix G.\n4\nSmall-scale Synthetic Supervision\nWe find that the PRM outperforms the ORM at large-scale, but this result alone\npaints an incomplete picture. To better compare outcome and process supervi-\nsion, there are two confounding factors that must be isolated. First, the training\nsets for the ORM and the PRM are not directly comparable: the PRM training\nset was constructed using active learning, is biased towards answer-incorrect\nsolutions, and is an order of magnitude smaller. Second, the final-answer grad-\ning will provide positive labels to spurious solutions that reach the correct final\nanswer despite incorrect reasoning. This could damage ORM performance, an\neffect we may or may not want to attribute to outcome supervision more gen-\nerally.\nDue to the high cost of collecting human feedback, we cannot easily ablate\nthese factors using human labelers. We instead perform the relevant ablations\nby using the large-scale PRM to supervise smaller models. This setup enables\nus to simulate a large amount of data collection at a modest cost.\nFor the\nremainder of this section, we refer to the large-scale PRM from Section 3 as\nPRMlarge.\n8\n\n\n4.1\nProcess vs Outcome Supervision\nWe now conduct a direct comparison of outcome and process supervision. We\nfirst sample between 1 and 200 solutions per problem from a small-scale genera-\ntor. For each dataset, we provide three forms of supervision: process supervision\nfrom PRMlarge, outcome supervision from PRMlarge, and outcome supervision\nfrom final-answer checking.\nThe choice of supervision is the only difference\nbetween these three series of reward models, which are otherwise trained on\nidentical datasets. See Appendix H for more details about how PRMlarge is\nused for outcome and process supervision.\nIn Figure 4a, we evaluate each reward model by its best-of-500 selection. We\nsee that process supervision significantly outperforms both forms of outcome\nsupervision at all data collection scales.\nIn Figure 4b, we evaluate the best\nreward model from each series by its best-of-N performance across different\nvalues of N. We see that using PRMlarge for outcome supervision is noticeably\nmore effective than final-answer checking. This can be explained by the fact\nthat PRMlarge provides better supervision for solutions that reach the correct\nfinal answer using incorrect reasoning.\nIt is not clear whether supervision by PRMlarge or by final-answer checking\nrepresents the more appropriate outcome supervision baseline.\nWhile final-\nanswer supervision is more explicitly outcome based, its main weakness — the\nexistence of false positives — is arguably over-emphasized in the MATH dataset.\nOutcome supervision by PRMlarge better represents outcome supervision in do-\nmains that are less susceptible to false positives. We consider outcome supervi-\nsion by PRMlarge to be the more relevant baseline, but we encourage the reader\nto draw their own conclusions.\n4.2\nActive Learning\nFinally, we investigate the impact of active learning. We train a small-scale\nreward model, PRMselector, on a single sample from each problem, and we use\nthis model to score 1000 samples per problem. To train each of our larger re-\nward models, we select N samples per problem such that 80% are the most\nconvincing (according to PRMselector) wrong-answer samples, and 20% are the\nmost convincing samples that remain (right- or wrong-answer). We score the\nselected samples with PRMlarge and train on those scores. This process ensures\nthat all samples are relatively convincing under PRMselector, that a large frac-\ntion are known to contain at least one mistake, and that our overall dataset\nis not too heavily biased toward wrong-answer solutions. Performance of this\ndata labelling scheme is shown in Figure 4a. By comparing the slopes of the\nline of best fit with and without active learning, we estimate that this form\nof active learning is approximately 2.6x more data efficient than uniform data\nlabelling. We note that the model trained on the largest active learning dataset\n(200 samples per problem) appears to slightly underperform the expected trend\nline. Our best explanation for this observation is that 200 samples represents\na significant fraction of the overall selection pool (1000 samples) and that this\n9\n\n\nORM\nPRM\nMajority Vote\n# Problems\nAP Calculus\n68.9%\n86.7%\n80.0%\n45\nAP Chemistry\n68.9%\n80.0%\n71.7%\n60\nAP Physics\n77.8%\n86.7%\n82.2%\n45\nAMC10/12\n49.1%\n53.2%\n32.8%\n84\nAggregate\n63.8%\n72.9%\n61.3%\n234\nTable 1: We measure out-of-distribution generalization using recent STEM tests.\nWe evaluate the outcome-supervised RM, the process-supervised RM, and ma-\njority voting using 100 test samples per problem.\nrelative lack of diversity limits the possible upside from active learning.\nWe also performed a preliminary investigation into the impact of iteratively\nretraining PRMselector throughout data collection. Between iterations, we re-\ntrained PRMselector using all currently labeled data. Unfortunately, we observed\ninstability in this process which we were unable to diagnose.\nThe resulting\nreward models performed no better than the models described above. We expect\nsome form of iterative retraining to be beneficial in active learning, but we\ncurrently have no concrete evidence to support this claim. We consider this a\ncompelling direction for future research.\n5\nOOD Generalization\nTo get some measure of out-of-distribution generalization, we evaluate our large-\nscale ORM and PRM on a held-out set of 224 STEM questions, pulled from the\nmost recent AP Physics, AP Calculus, AP Chemistry, AMC10, and AMC12 ex-\nams. Since these tests were released after the pre-training dataset was compiled,\nwe can have high confidence that the model has not seen these problems. We\nreport the best-of-100 performance of the ORM, PRM and majority voting in\nTable 1. We observe results similar to those in Section 3: the PRM outperforms\nboth the ORM and majority voting. This shows us that the PRM can tolerate\na modest amount of distribution shift and that its strong performance holds up\non fresh test questions.\n6\nDiscussion\n6.1\nCredit Assignment\nOne clear advantage of process supervision is that it provides more precise\nfeedback than outcome supervision.\nA reward model trained with outcome\nsupervision faces a difficult credit-assignment task — to generalize well, it must\ndetermine where an incorrect solution went wrong. This is particularly difficult\nfor hard problems: most model-generated solutions contain an error somewhere,\nso the marginal value of a negative label from outcome supervision is low. In\n10\n\n\ncontrast, process supervision provides a richer signal: it specifies both how\nmany of the first steps were in fact correct, as well as the precise location of\nthe incorrect step. Process supervision makes credit assignment easier, and we\nbelieve that this explains its strong performance.\n6.2\nAlignment Impact\nProcess supervision has several advantages over outcome supervision related\nto AI alignment. Process supervision is more likely to produce interpretable\nreasoning, since it encourages models to follow a process endorsed by humans.\nProcess supervision is also inherently safer: it directly rewards an aligned chain-\nof-thought rather than relying on outcomes as a proxy for aligned behavior\n(Stuhlm¨\nuller and Byun, 2022). In contrast, outcome supervision is harder to\nscrutinize, and the preferences conveyed are less precise.\nIn the worst case,\nthe use of outcomes as an imperfect proxy could lead to models that become\nmisaligned after learning to exploit the reward signal (Uesato et al., 2022; Cotra,\n2022; Everitt et al., 2017).\nIn some cases, safer methods for AI systems can lead to reduced performance\n(Ouyang et al., 2022; Askell et al., 2021), a cost which is known as an alignment\ntax. In general, any alignment tax may hinder the adoption of alignment meth-\nods, due to pressure to deploy the most capable model. Our results show that\nprocess supervision in fact incurs a negative alignment tax. This could lead to\nincreased adoption of process supervision, which we believe would have positive\nalignment side-effects. It is unknown how broadly these results will generalize\nbeyond the domain of math, and we consider it important for future work to\nexplore the impact of process supervision in other domains.\n6.3\nTest Set Contamination\nThe test set of the MATH dataset contains problems that are discussed in\nseveral online venues, and it is likely that some of these problems appear in\nthe pretraining dataset for our models. We attempted to remove all MATH\nproblems from our MathMix dataset using string-matching heuristics, but since\nhumans can post hard-to-detect rephrasings of a problem online, it is difficult\nto make any strong guarantees about the overlap between MathMix and the\nMATH dataset.\nIn our experience inspecting model-generated solutions, we saw no clear signs\nof our models memorizing MATH problems. However, it is impossible to rule\nout subtle forms of memorization that would slip past manual inspection, and\nit is still possible that some degree of contamination has slightly inflated our\nperformance on the MATH test set. Even in that case, we would expect any\ncontamination to manifest similarly across all methods, and that the relative\ncomparisons made throughout this work would remain mostly unaffected.\nWe also note that the PRM regularly surfaces correct solutions to MATH\nproblems that have a low single-digit percentage solve-rate under the genera-\ntor, some examples of which can be seen in Appendix I. The generator’s low\n11\n\n\nsolve-rate is an additional indication that it has not encountered such problems\nvia test set contamination. Our generalization results from Section 5 further\nstrengthen our claim that test set contamination has not significantly impacted\nthis work, since we observe qualitatively similar results on problems that are\nguaranteed to be uncontaminated.\n7\nRelated Work\n7.1\nOutcome vs Process Supervision\nIn work closely related to our own, Uesato et al. (2022) compare the impact\nof outcome and process supervision in the domain of grade school math. They\nfound that both methods led to similar final-answer error rates, and that process\nsupervision achieved those results with less data. While our core methodology is\nvery similar, there are three main details that differ. First, we use a more capable\nmodel to collect PRM800K dataset and to perform our large-scale experiments.\nHowever, our small-scale results in Section 4 suggest that large-scale models are\nnot necessary to observe benefits from process supervision. Second, we evaluate\non the MATH dataset, which is significantly more challenging than GSM8K.\nThird, we collect a much larger quantity of process supervision data.\nOn the surface, the results from Uesato et al. (2022) may seem to conflict\nwith our claim that process supervision leads to better performance. However,\nwe believe the apparent conflict can be explained by the difference in the scale\nof the supervision. The data scaling trend in Figure 4a suggests that a small\namount of process supervision and a large amount of outcome supervision do\nin fact lead to similar performance, consistent with the results from Uesato\net al. (2022).\nThe trend also shows that process supervision beats outcome\nsupervision when scaled up, even when judged based solely on outcomes. This\nis consistent with our results in Section 3.\nWe believe these results make a\nstrong case for using process supervision.\n7.2\nSynthetic Supervision\nSimilar to our work in Section 4, Gao et al. (2022) use a large reward model to\nsupervise the training of smaller models. They study the over-optimization that\noccurs during RLHF, with experiments that require large quantities of human\npreference data. To work around this challenge, they use a gold-standard reward\nmodel to replace human feedback. Our use of a large-scale reward model to\nsupervise smaller reward models shares similarities with their approach.\n7.3\nNatural Language Reasoning\nSeveral recent studies that have examined the reasoning ability of large language\nmodels are implicitly relevant to our work. Lewkowycz et al. (2022) showed that\nfinetuning models on a large corpus of technical content led to significantly im-\nproved performance on MATH. Wang et al. (2022) showed that self-consistency\n12\n\n\nleads to remarkably strong performance on many reasoning benchmarks, no-\ntably without requiring any additional finetuning. Wei et al. (2022) and Nye\net al. (2021) demonstrate the importance of explicitly performing intermediate\nreasoning steps via a chain of thought or a scratchpad in order to solve tasks\nthat require multi-step reasoning. Kojima et al. (2022) show that models are\nable to perform this behavior zero-shot, conditioned only on a simple prompt.\n8\nConclusion\nWe have shown that process supervision can be used to train much more reliable\nreward models than outcome supervision in the domain of mathematical rea-\nsoning. We have also shown that active learning can be used to lower the cost of\nhuman data collection by surfacing only the most valuable model completions\nfor human feedback. We release PRM800K, the full dataset of human feedback\nused to train our state-of-the-art reward model, with the hope that removing\nthis significant barrier to entry will catalyze related research on the alignment of\nlarge language models. We believe that process supervision is currently under-\nexplored, and we are excited for future work to more deeply investigate the\nextent to which these methods generalize.\nAcknowledgements\nWe thank Joshua Achiam, Mark Chen, Jonathan Gordon, Dan Hendrycks,\nLukasz Kaiser, Oleg Murk, Ben Sokolowsky, Francis Song, and Jonathan Uesato\nfor valuable feedback and thoughtful discussions; Giambattista Parascandolo\nand Daniel Selsam for their contributions to the MathMix dataset; Jonathan\nWard for contributing to the data collection interface; Wojciech Zaremba for en-\ncouraging us to scale up data collection; Peter Hoeschele and Aris Kostantinidis\nfor supporting our data collection; the research acceleration and supercomput-\ning teams at OpenAI for providing infrastructure support; and the team at Scale\nand the many data-labelers who created PRM800K.\nReferences\nA. Askell, Y. Bai, A. Chen, D. Drain, D. Ganguli, T. Henighan, A. Jones,\nN. Joseph, B. Mann, N. DasSarma, et al. A general language assistant as a\nlaboratory for alignment. arXiv preprint arXiv:2112.00861, 2021.\nS. Bubeck, V. Chandrasekaran, R. Eldan, J. Gehrke, E. Horvitz, E. Kamar,\nP. Lee, Y. T. Lee, Y. Li, S. Lundberg, et al.\nSparks of artificial general\nintelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712,\n2023.\n13\n\n\nP. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei.\nDeep reinforcement learning from human preferences.\nAdvances in neural\ninformation processing systems, 30, 2017.\nK. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert,\nJ. Tworek, J. Hilton, R. Nakano, et al. Training verifiers to solve math word\nproblems. arXiv preprint arXiv:2110.14168, 2021.\nA.\nCotra.\nWithout\nspecific\ncountermeasures,\nthe\neasi-\nest\npath\nto\ntransformative\nAI\nlikely\nleads\nto\nAI\ntakeover.\nhttps://www.alignmentforum.org/posts/pRkFkzwKZ2zfa3R6H/\nwithout-specific-countermeasures-the-easiest-path-to, 2022.\nA. Creswell, M. Shanahan, and I. Higgins.\nSelection-inference:\nExploiting\nlarge language models for interpretable logical reasoning.\narXiv preprint\narXiv:2205.09712, 2022.\nT. Everitt, V. Krakovna, L. Orseau, M. Hutter, and S. Legg. Reinforcement\nlearning with a corrupted reward channel. arXiv preprint arXiv:1705.08417,\n2017.\nL. Gao, J. Schulman, and J. Hilton. Scaling laws for reward model overopti-\nmization. arXiv preprint arXiv:2210.10760, 2022.\nD. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song,\nand J. Steinhardt. Measuring mathematical problem solving with the math\ndataset. arXiv preprint arXiv:2103.03874, 2021.\nT. Kojima, S. S. Gu, M. Reid, Y. Matsuo, and Y. Iwasawa. Large language\nmodels are zero-shot reasoners. arXiv preprint arXiv:2205.11916, 2022.\nA. Lewkowycz, A. Andreassen, D. Dohan, E. Dyer, H. Michalewski, V. Ra-\nmasesh, A. Slone, C. Anil, I. Schlag, T. Gutman-Solo, et al.\nSolving\nquantitative reasoning problems with language models.\narXiv preprint\narXiv:2206.14858, 2022.\nY. Li, Z. Lin, S. Zhang, Q. Fu, B. Chen, J.-G. Lou, and W. Chen.\nOn\nthe advance of making language models better reasoners.\narXiv preprint\narXiv:2206.02336, 2022.\nJ. Maynez, S. Narayan, B. Bohnet, and R. McDonald.\nOn faithfulness and\nfactuality in abstractive summarization.\narXiv preprint arXiv:2005.00661,\n2020.\nR. Nakano, J. Hilton, S. Balaji, J. Wu, L. Ouyang, C. Kim, C. Hesse, S. Jain,\nV. Kosaraju, W. Saunders, et al.\nWebgpt:\nBrowser-assisted question-\nanswering with human feedback. arXiv preprint arXiv:2112.09332, 2021.\nE. Nichols, L. Gao, and R. Gomez. Collaborative storytelling with large-scale\nneural language models. In Proceedings of the 13th ACM SIGGRAPH Con-\nference on Motion, Interaction and Games, pages 1–10, 2020.\n14\n\n\nM. Nye, A. J. Andreassen, G. Gur-Ari, H. Michalewski, J. Austin, D. Bieber,\nD. Dohan, A. Lewkowycz, M. Bosma, D. Luan, et al.\nShow your work:\nScratchpads for intermediate computation with language models.\narXiv\npreprint arXiv:2112.00114, 2021.\nOpenAI. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.\nL. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin,\nC. Zhang, S. Agarwal, K. Slama, A. Ray, et al. Training language models to\nfollow instructions with human feedback. arXiv preprint arXiv:2203.02155,\n2022.\nJ. Shen, Y. Yin, L. Li, L. Shang, X. Jiang, M. Zhang, and Q. Liu. Generate\n& rank: A multi-task framework for math word problems. arXiv preprint\narXiv:2109.03034, 2021.\nN. Stiennon, L. Ouyang, J. Wu, D. Ziegler, R. Lowe, C. Voss, A. Radford,\nD. Amodei, and P. F. Christiano. Learning to summarize with human feed-\nback.\nAdvances in Neural Information Processing Systems, 33:3008–3021,\n2020.\nA. Stuhlm¨\nuller and J. Byun. Supervise process, not outcomes. https://ought.\norg/updates/2022-04-06-process, 2022.\nJ. Uesato, N. Kushman, R. Kumar, F. Song, N. Siegel, L. Wang, A. Creswell,\nG. Irving, and I. Higgins.\nSolving math word problems with process-and\noutcome-based feedback. arXiv preprint arXiv:2211.14275, 2022.\nX. Wang, J. Wei, D. Schuurmans, Q. Le, E. Chi, and D. Zhou. Self-consistency\nimproves chain of thought reasoning in language models.\narXiv preprint\narXiv:2203.11171, 2022.\nJ. Wei, X. Wang, D. Schuurmans, M. Bosma, E. Chi, Q. Le, and D. Zhou.\nChain of thought prompting elicits reasoning in large language models. arXiv\npreprint arXiv:2201.11903, 2022.\nE. Zelikman, Y. Wu, J. Mu, and N. Goodman. Star: Bootstrapping reason-\ning with reasoning. Advances in Neural Information Processing Systems, 35:\n15476–15488, 2022.\nD. M. Ziegler, N. Stiennon, J. Wu, T. B. Brown, A. Radford, D. Amodei,\nP. Christiano, and G. Irving. Fine-tuning language models from human pref-\nerences. arXiv preprint arXiv:1909.08593, 2019.\n15\n\n\nA\nMathMix\nSimilar to Lewkowycz et al. (2022) we construct a large-scale dataset of high-\nquality math-relevant tokens for use in a lightweight pretraining stage, before\nfinetuning on comparably smaller datasets like MATH and PRM800K. This\ndataset, which we call MathMix, has two main differences compared to the one\nused to train Minerva. First, it is smaller and more aggressively filtered to high-\nquality math problem-solving content, and second, it does not explicitly mix in\ngeneral language data.\nMinerva was trained on 38.5B tokens of arXiv documents and webscrape\npages with LaTeX content, while MathMix consists of a smaller set of 1.5B\ntokens containing individual math problems and their solutions, free-form text\ndiscussing math problems and concepts, and synthetic data (Table 2). While\nMinerva was pretrained on a dataset with 5% general natural language data,\nwe chose not to mix in any natural language data explicitly, primarily because\nMathMix already contains plenty of natural language data.\nData type\nToken count\nPresent in pretraining?\nMath problems and solutions\n∼275M\nNo\nFree-form math discussion text (1)\n∼430M\nNo\nFree-form math discussion text (2)\n∼450M\nYes\nSynthetic data (1)\n∼30M\nNo\nSynthetic data (2)\n∼100M\nYes\nCritiques grading data\n∼500M\nNo\nTable 2: MathMix dataset components.\nNote that when training smaller models, as in Section 4, we use a slightly\nsmaller variant of MathMix that excludes the critiques data and only consists of\n1B tokens. For our large models experiments, we train on MathMix for roughly\n3B tokens (2 epochs). For our small models experiments, we train for 6 epochs\n(roughly 6.6B tokens).\nWe apply a set of decontamination checks on MathMix against the test\nsplit of the MATH dataset, including stripping out LaTeX and searching for\nmatching n-grams, but we can make no strong guarantees on the efficacy of this\ndecontamination. As discussed in Section 6.3, we would not expect the relative\ncomparisons made throughout this work to be significantly impacted by test set\ncontamination.\n16\n\n\nB\nPRM800K\nWe collected 1,085,590 step-level labels over 101,599 solution samples.\nWe\npresent the whole unfiltered dataset as PRM800K. During training we discard\nlabels used for quality control, as well as any step-level labels for which the\nlabeler was unable to complete the task. The filtered dataset contains about\n800,000 step-level labels over 75,000 solutions. The full PRM800K dataset is\navailable at https://github.com/openai/prm800k.\nThe data collection was split into two separate phases. In phase 1, we col-\nlected labels for multiple alternative completions at each step of a solution. This\nseeded our dataset but was cumbersome—for many steps the alternatives were\nrepetitive, and we found labelers spent a lot of time supervising long uninter-\nesting solutions. As a result, the step-level labels we collected in this phase are\nmore repetitive than those collected later. In total, phase 1 represents about\n5% of PRM800K, or about 40,000 step-level labels.\nThe majority of our labels were collected as part of phase 2, during which we\nscaled up and streamlined the data collection process. Phase 2 data collection\nis split into 10 generations. For each generation, we sample N solutions per\nproblem from the generator.\nWe rank these solutions with our current best\nPRM and surface the highest scoring wrong-answer solutions to our labelers.\nWe retrain this PRM between each generation using all the latest data. This\nactive learning strategy changes the balance of our data considerably. Though\nwe sometimes surfaced correct solutions (either by manually injecting correct\nsolutions or because of errors in our automatic grading), the vast majority of the\nlabels we collected in this phase are for incorrect solutions. Table 3 breaks down\nthe balance of correct/incorrect steps and solutions between the different phases\nof data collection. Though we mostly collected labels on incorrect solutions, we\nstill collected many labels for correct individual steps. In fact, our small-scale\nablations in Section 4.2 suggest that this active learning strategy, which favors\nlabelling high-scoring wrong-answer solutions, improves performance despite the\nresulting imbalance in the dataset.\nphase 1\nphase 2\ncombined\n% end in correct solution\n85.1\n13.2\n14.2\n% correct steps\n58.6\n74.1\n73.1\nTable 3: Distribution of positive/negative steps/solutions.\nSome of our phase 2 questions are intended for quality control. For a quality\ncontrol question, researchers mark which steps are reasonable to label as in-\ncorrect. Then we assess that labelers are able to consistently mark those steps\nas incorrect. Prior to starting on phase 2, we required all labelers to label 30\nquality control questions. This served as a screening test, and we only admitted\nlabelers that agreed with our gold labels at least 75% of the time.\nWe then designated 10-20 problems per generation as additional quality\ncontrol questions, and we randomly served them to labelers as they worked\n17\n\n\nthrough the task.\nWe used the results of this continuous quality control to\nremove labelers whose quality slipped too far, as well as to prepare educational\nmaterial on common mistakes in order to improve labeler alignment with our\ninstructions.\nC\nEvaluation\nAs we scaled up the project, we began having to collect labels on multiple\nsolutions for the same training problem. In order to avoid the risk of over-fitting\non the 7,500 MATH training problems, we expanded the training set to include\n4,500 MATH test split problems. We therefore evaluate our models only on the\nremaining 500 held-out problems. We selected these 500 test problems uniformly\nat random. In Figure 5, we show that the distribution of difficulty levels and\nsubjects in this subset is representative of the MATH test set as a whole. The\nspecific test set we used can be found at https://github.com/openai/prm800k.\nWe leave it for future work to explore how many distinct training problems are\nactually necessary, and how quickly our methods overfit to the training set.\nFigure 5: Two histograms comparing the distribution of problem difficulty levels\nand subjects in both the original MATH test set and in our 500 problem test\nsubset.\n18\n\n\nD\nLabelling Instructions\nLabelers were tasked to look at steps in a solution and label each one as posi-\ntive, negative, or neutral. A step is considered neutral if it is appropriate in\ncontext, reasonable, correct, and contains only computations that can be veri-\nfied easily. A step is positive if it is neutral and also progresses towards the\nsolution. All other steps are considered negative. Labelers were not given ref-\nerence solutions, but they were given the ground truth final answers. We chose\nnot to provide reference solutions to avoid biasing them towards one particular\npath to the solution. We chose to provide ground truth final answers since this\ninformation can sometimes help labelers resolve their own misunderstandings.\nIn phase 1, labelers were permitted to enter their own steps in the case that\nall candidate steps were negative. Then the solution would progress from a\nrandomly selected positive step (or neutral if their were no positive ones).\nThis often resulted in trajectories that got stuck in endless sequences of neutral\nsteps that said reasonable things but made frustratingly slow progress towards a\nsolution or negative steps that needed constant human supervision. In phase 2,\nwe pre-generate whole solutions and end the task as soon as the first negative\nstep is encountered.\nThe full instructions given to labelers can be found at\nhttps://github.com/openai/prm800k/tree/main/prm800k/instructions.\nE\nORM Training Details\nWe train outcome-supervised reward models in the same manner as token-level\nverifiers from Cobbe et al. (2021), with a few subtle differences to hyperparam-\neters. In particular, we only train for a single epoch on each dataset of model\nsamples and reward model labels, without dropout, and without jointly learn-\ning a language modeling objective. We find that performance is not sensitive to\nmost other hyperparameters, within a reasonable range.\nTo collect model samples, we simply sample uniformly from the generator at\na temperature of 1.0 without applying any rebalancing of positives or negatives.\nAt training time, the reward model makes predictions for every token in the\ncontext. The target for each token in a solution is the same, based on whether\nthe solution is labelled correct or incorrect. At test time, we simply use the\nscore of the final token in the completion as the overall score of the solution.\nWe note that this setup is identical to the way token-level verifiers were trained\nin Cobbe et al. (2021).\n19\n\n\nF\nPRM Details\nF.1\nTraining\nWe train our PRMs by fine-tuning the MathMix model to predict the probability\nof positive, negative, and neutral labels given a solution prefix ending in one of\nour labeled steps. We sweep over hyperparameters using a dataset containing\nthe first ∼10% of PRM800K. Fine-tuning an LLM from its ordinary language\nmodeling task to a classification task like this is a large distribution shift, and\nwe found low learning rates were important to stable PRM training.\nAll of our PRMs are trained for 2 epochs. On smaller datasets (such as\nin phase 1 and the first few generations of phase 2) this improves the final\nperformance over training for just 1 epoch. Additional epochs, up to some point,\ndon’t noticeably help or hurt performance. On larger datasets, the benefits of\n2 epoch training diminishes, but we continue doing it for consistency.\nF.2\nScoring\nThere are multiple ways of using the PRM to score solutions. In general, we\nproduce a single solution-level score by performing a reduction over step-level\nscores, where the step-level score is the probability that the step’s label is pos-\nitive. This involves two specific implementation decisions. First, when deter-\nmining a step-level score, we either consider a neutral label to be positive or\nnegative. Second, when determining a solution-level score, we either use the\nminimum or the product over step-level scores as a reduction.\nWe show results from all four scoring strategies in Table 4. The best per-\nforming strategy is to take the product of step-level scores and to consider the\nneutrals as positives, but the difference in performance between all strategies\nis minor. Throughout the rest of this work, we consider neutral steps to be\npositive, and we define the solution score to be the product of step-level scores.\nUsing the product instead of the minimum as the reduction does create a slight\nbias against solutions with a larger number of steps.\nproduct\nminimum\nneutral = positive\n78.2%\n77.6%\nneutral = negative\n77.4%\n77.8%\nTable 4: Best-of-1860 test performance using the PRM with four different scor-\ning strategies.\n20\n\n\nG\nDifficulty Breakdown\nWe show performance of our ORM and PRM on each quintile of the MATH\ndataset. We determine quintiles based on the pass rate under the generator.\nIt is interesting to note that the performance gap is not only apparent on high\ndifficulty problems: it is in fact apparent across all difficulties. For the lowest\ndifficulty problems, we see that it is possible to find adversarial examples that\nfool the ORM, since the ORM’s performance slightly decreases as the number of\nsamples increases. In contrast, the PRM remains highly robust over this same\nset of samples.\nWe also see that increasing the number of samples has the largest positive\neffect on the highest difficulty problems. This is to be expected, since a large\nnumber of generator samples may be required to find a true and convincing\nsolution to a hard problem.\n101\n103\n0.0\n0.2\n0.4\n0.6\n0.8\n1.0\n% Problems Solved (Best-of-N)\nQuintile 1 (easiest)\nPRM\nORM\n101\n103\nQuintile 2\n101\n103\nN = number of solutions per problem\nQuintile 3\n101\n103\nQuintile 4\n101\n103\nQuintile 5 (hardest)\nFigure 6: A breakdown of ORM vs PRM performance by problem difficulty.\n21\n\n\nH\nSynthetic Supervision Details\nWe can use PRMlarge to provide either outcome or process supervision for\nsmaller models.\nWe determine the labels for individual steps based on the\nstep-level probabilities outputted by PRMlarge. To do this, we set an arbitrary\nthreshold: any step that PRMlarge assigns a negative label with greater than\n20% probability is considered incorrect. We choose this threshold based on the\nobservation that PRMlarge is slightly miscalibrated in the direction of favoring\npositive labels.\nTo provide process supervision for a solution, we directly return the step-\nlevel labels (positive or negative) provided by PRMlarge, up until the first step\nthat is marked as negative. This mimics our true human data collection process.\nTo provide outcome supervision, we mark the solution as correct if and only if\nPRMlarge considers every step to be correct (using the same thresholding logic).\n22\n\n\nI\nPRM Visualizations\nAll examples shown come from the large-scale generator (GPT-4).\nWe note\nthe pass-rate under the generator to give some sense of the difficulty of these\nproblems.\nI.1\nTrue Positives\nThese cherry-picked examples show the best-of-1860 solution from the generator\nas ranked by the large-scale PRM.\nProblem 1. Generator pass-rate: 0.1%. This challenging trigonometry problem\nrequires applying several identities in a not-at-all obvious succession.\nMost\nsolution attempts fail, because it is hard to choose which identities are actually\nhelpful. Though successful solutions to this problem are rare, the reward model\ncorrectly recognizes when a valid chain-of-thought has been found.\n23\n\n\nProblem 2. Generator pass-rate: 5.8%. In step 7 and 8, the generator starts\nperforming guess-and-check. This is a common place the model might hallu-\ncinate, by claiming a particular guess is successful when it isn’t. In this case,\nthe reward model verifies each step and determines that the chain-of-thought is\ncorrect.\nProblem 3. Generator pass-rate: 1.7%. The generator successfully applies sev-\neral trigonometric identities to simplify the expression.\n24\n\n\nProblem 4. Generator pass-rate: 4.5%. Here, the generator successfully per-\nforms a complex series of polynomial factorizations.\nThe use of the Sophie-\nGermain identity in step 5 is an important step that could be considered in-\nsightful.\nI.2\nTrue Negatives\nProblem 5.\nGenerator pass-rate: 4.5%.\nThe generator attempts to use the\ndifference of squares formula in step 12 on an expression that isn’t in fact a\ndifference of squares. The reward model catches this mistake.\n25\n\n\nProblem 6. Generator pass-rate: 93.5%. In step 7, the generator makes an\nincorrect attempt to simplify an expression.\nThe reward model catches this\nmistake.\nProblem 7. Generator pass-rate: 48.0%. In step 11, the generator makes a\nsimple calculation error. The reward model catches this mistake.\n26\n\n\nProblem 8. Generator pass-rate: 5.8%. The justification in step 8 is strange,\nbut the reward model lets it slide. In step 9, though, the model incorrectly\nfactors the expression. The reward model catches this mistake.\nI.3\nFalse Positives\nProblem 9. Generator pass-rate: 18.5%. The generator makes a subtle counting\nerror in step 9. On the surface, it appears reasonable to claim that there are 5\nways to exchange the same colored ball since there are 5 colors. However, this\nundercounts by a factor of 2, since Bob has 2 choices for which ball to return\nto Alice. The reward model is fooled by this mistake.\n27\n\n\nProblem 10. Generator pass-rate: 17.6%. In step 13, the generator attempts\nto simplify the equation by combining like terms. It correctly moves and com-\nbines the linear terms to the left-hand side, but then mistakenly leaves the\nright-hand side untouched. The reward model is fooled by this mistake.\nProblem 11. Generator pass-rate: 13.4%. The generator attempts to per-\nform long division, but in step 16, it forgets to include the leading zeros in the\nrepeating part of the decimal. The reward model is fooled by this mistake.\n28\n\n\nProblem 12. Generator pass-rate: 9.1%. In step 4, the generator falsely\nclaims that the sequence repeats itself every 12 terms, when it’s in fact every\n10 terms. This sort of counting mistake occasionally fools the reward model.\n29\n\n\nMATH-SHEPHERD: VERIFY AND REINFORCE LLMS\nSTEP-BY-STEP WITHOUT HUMAN ANNOTATIONS\nPeiyi Wang1†\nLei Li3\nZhihong Shao4\nR.X. Xu2\nDamai Dai1\nYifei Li5\nDeli Chen2\nY. Wu2\nZhifang Sui1\n1National Key Laboratory for Multimedia Information Processing, Peking University\n2DeepSeek-AI\n3The University of Hong Kong\n4Tsinghua University\n5The Ohio State University\n{wangpeiyi9979, nlp.lilei}@gmail.com\nli.14042@osu.edu\nszf@pku.edu.cn\nProject Page: MA T H-SH E P H E R D\nABSTRACT\nIn this paper, we present an innovative process-oriented math process reward model\ncalled MATH-SHEPHERD, which assigns a reward score to each step of math\nproblem solutions. The training of MATH-SHEPHERD is achieved using automati-\ncally constructed process-wise supervision data, breaking the bottleneck of heavy\nreliance on manual annotation in existing work. We explore the effectiveness of\nMATH-SHEPHERD in two scenarios: 1) Verification: MATH-SHEPHERD is utilized\nfor reranking multiple outputs generated by Large Language Models (LLMs);\n2) Reinforcement Learning: MATH-SHEPHERD is employed to reinforce LLMs\nwith step-by-step Proximal Policy Optimization (PPO). With MATH-SHEPHERD, a\nseries of open-source LLMs demonstrates exceptional performance. For instance,\nthe step-by-step PPO with MATH-SHEPHERD significantly improves the accuracy\nof Mistral-7B (77.9%→84.1% on GSM8K and 28.6%→33.0% on MATH). The\naccuracy can be further enhanced to 89.1% and 43.5% on GSM8K and MATH\nwith the verification of MATH-SHEPHERD, respectively. We believe that automatic\nprocess supervision holds significant potential for the future evolution of LLMs.\nLLaMA2-70B\nMAmmoTH\nLLaMA2-70B\nWizardMATH\nLLaMA2-70B\nMetaMATH*\nLLemma-34B\nMetaMATH*\nDeepSeek-67B\nMetaMATH*\nGSM8K\n70\n75\n80\n85\n90\n95\nAccuracy (%)\n72.4\n81.6\n93.2\n90.9\n93.3\n72.4\n81.6\n80.4\n75.8\n82.8\nGPT-4 (early): 92.0\nGPT-4-0613*: 94.4\nFine-tuned LLMs\n+SHEPHERD\nLLaMA2-70B\nMAmmoTH\nLLaMA2-70B\nWizardMATH\nLLaMA2-70B\nMetaMATH*\nLLemma-34B\nMetaMATH*\nDeepSeek-67B\nMetaMATH*\nMATH\n10\n20\n30\n40\n50\n60\nAccuracy (%)\n21.1\n22.7\n45.2\n47.3\n48.1\n21.1\n22.7\n29.8\n34.8\n36.8\nGPT-4 (early): 42.5\nGPT-4-0613*: 56.2\nFine-tuned LLMs\n+SHEPHERD\nFigure 1: We evaluate the performance of various LLMs with MATH-SHEPHERD on the GSM8K\nand MATH datasets. All base models are finetuned with the MetaMath dataset (Yu et al., 2023b).\nThe +SHEPHERD results are obtained by selecting the best one from 256 candidates using MATH-\nSHEPHERD. We observe that MATH-SHEPHERD is compatible with different LLMs. The results of\nGPT-4 (early) are from Bubeck et al. (2023).\n†Contribution during internship at DeepSeek-AI.\narXiv:2312.08935v3  [cs.AI]  19 Feb 2024\n\n\n1\nINTRODUCTION\nLarge language models (LLMs) have demonstrated remarkable capabilities across various tasks\n(Park et al., 2023; Kaddour et al., 2023; Song et al.; Li et al., 2023a; Wang et al., 2023a; Chen\net al., 2023; Zheng et al., 2023; Wang et al., 2023c), However, even the most advanced LLMs face\nchallenges in complex multi-step mathematical reasoning problems (Lightman et al., 2023; Huang\net al., 2023). To address this issue, prior research has explored different methodologies, such as pre-\ntraining (Azerbayev et al., 2023), fine-tuning (Luo et al., 2023; Yu et al., 2023b; Wang et al., 2023b),\nprompting (Wei et al., 2022; Fu et al., 2022), and verification (Wang et al., 2023d; Li et al., 2023b;\nZhu et al., 2023; Leviathan et al., 2023). Among these techniques, verification has recently emerged\nas a favored method. The motivation behind verification is that relying solely on the top-1 result may\nnot always produce reliable outcomes. A verification model can rerank candidate responses, ensuring\nhigher accuracy and consistency in the outputs of LLMs. In addition, a good verification model can\nalso offer invaluable feedback for further improvement of LLMs (Uesato et al., 2022; Wang et al.,\n2023b; Pan et al., 2023).\nThe verification models generally fall into the outcome reward model (ORM) (Cobbe et al., 2021;\nYu et al., 2023a) and process reward model (PRM) (Li et al., 2023b; Uesato et al., 2022; Lightman\net al., 2023; Ma et al., 2023). The ORM assigns a confidence score based on the entire generation\nsequence, whereas the PRM evaluates the reasoning path step-by-step. PRM is advantageous due to\nseveral compelling reasons. One major benefit is its ability to offer precise feedback by identifying\nthe specific location of any errors that may arise, which is a valuable signal in reinforcement learning\nand automatic correction. Besides, The PRM exhibits similarities to human behavior when assessing\na reasoning problem. If any steps contain an error, the final result is more likely to be incorrect,\nmirroring the way human judgment works. However, gathering data to train a PRM can be an arduous\nprocess. Uesato et al. (2022) and Lightman et al. (2023) utilize human annotators to provide process\nsupervision annotations, enhancing the performance of PRM. Nevertheless, annotation by humans,\nparticularly for intricate multi-step reasoning tasks that require advanced annotator skills, can be\nquite costly, which hinders the advancement and practical application of PRM.\nTo tackle the problem, in this paper, we propose an automatic process annotation framework. In-\nspired by Monte Carlo Tree Search (Kocsis & Szepesv´\nari, 2006; Coulom, 2006; Silver et al., 2016;\n´\nSwiechowski et al., 2023), we define the quality of an intermediate step as its potential to deduce\nthe correct final answer. By leveraging the correctness of the answer, we can automatically gather\nstep-wise supervision. Specifically, given a math problem with a golden answer and a step-by-step\nsolution, to achieve the label of a specific step, we utilize a fine-tuned LLM to decode multiple\nsubsequent reasoning paths from this step. We further validate whether the decoded final answer\nmatches with the golden answer. If a reasoning step can deduce more correct answers than another, it\nwould be assigned a higher correctness score.\nWe use this automatic way to construct the training data for MATH-SHEPHERD, and verify our ideas\non two widely used mathematical benchmarks, GSM8K (Cobbe et al., 2021) and MATH (Hendrycks\net al., 2021). We explore the effectiveness of MATH-SHEPHERD in two scenarios: 1) verification:\nMATH-SHEPHERD is utilized for reranking multiple outputs generated by LLMs; 2) reinforcement\nlearning: MATH-SHEPHERD is employed to reinforce LLMs with step-by-step Proximal Policy\nOptimization (PPO). With the verification of MATH-SHEPHERD, a series of open-source LLMs from\n7B to 70B demonstrates exceptional performance. For instance, the step-by-step PPO with MATH-\nSHEPHERD significantly improves the accuracy of Mistral-7B (77.9%→84.1% on GSM8K and\n28.6%→33.0% on MATH). The accuracy can be further enhanced to 89.1% and 43.5% on GSM8K\nand MATH with verification. DeepSeek 67B (DeepSeek, 2023) achieves accuracy rates of 93.3% on\nthe GSM8K dataset and 48.1% on the MATH dataset with verification of MATH-SHEPHERD. To the\nbest of our knowledge, these results are unprecedented for open-source models that do not rely on\nadditional tools.\nOur main contributions are as follows:\n1) We propose a framework to automatically construct process supervision datasets without human\nannotations for math reasoning tasks.\n\n\n2) We evaluate our method on both step-by-step verification and reinforcement learning scenarios.\nExtensive experiments on two widely used mathematical benchmarks - GSM8K and MATH, in\naddition to a series of LLMs ranging from 7B to 70B, demonstrate the effectiveness of our method.\n3) We empirically analyze the key factors for training high-performing process reward models,\nshedding light on future directions toward improving reasoning capability with automatic step-by-\nstep verification and supervision.\n2\nRELATED WORKS\nImproving and eliciting mathematical reasoning abilities of LLMs.\nMathematical reasoning\ntasks are one of the most challenging tasks for LLMs. Researchers have proposed various methods to\nimprove or elicit the mathematical reasoning ability of LLMs, which can be broadly divided into three\ngroups: 1) pre-training: The pre-training methods (OpenAI, 2023; Anil et al., 2023; Touvron et al.,\n2023; Azerbayev et al., 2023) pre-train LLMs on a vast of datasets that are related to math problems,\nsuch as the Proof-Pile and ArXiv (Azerbayev et al., 2023) with a simple next token prediction\nobjective. 2) fine-tuning: The fine-tuning methods (Yu et al., 2023b; Luo et al., 2023; Yue et al., 2023;\nWang et al., 2023b; Gou et al., 2023) can also enhance the mathematical reasoning ability of LLMs.\nThe core of fine-tuning usually lies in constructing high-quality question-response pair datasets with\na chain-of-thought reasoning process. and 3) prompting: The prompting methods (Wei et al., 2022;\nZhang et al., 2023; Fu et al., 2022; Bi et al., 2023) aim to elicit the mathematical reasoning ability\nof LLMs by designing prompting strategy without updating the model parameters, which is very\nconvenient and practical.\nMathematical reasoning verification for LLMs.\nExcept for directly improving and eliciting the\nmathematical reasoning potential of LLMs, the reasoning results can be boosted via an extra verifier\nfor selecting the best answer from multiple decoded candidates. There are two primary types of\nverifiers: the Outcome Reward Model (ORM) and the Process Reward Model (PRM). The ORM\nallocates a score to the entire solution while the PRM assigns a score to each individual step in\nthe reasoning process. Recent findings by (Lightman et al., 2023) suggest that PRM outperforms\nORM. In addition to verification, reward models can offer invaluable feedback for further training of\ngenerators (Uesato et al., 2022; Pan et al., 2023). Compared to ORM, PRM provides more detailed\nfeedback, demonstrating greater potential to enhance generator (Wu et al., 2023). However, training a\nPRM requires access to expensive human-annotated datasets (Uesato et al., 2022; Lightman et al.,\n2023), which hinders the advancement and practical application of PRM. Therefore, in this paper,\nwe aim to build a PRM for mathematical reasoning without human annotation, and we explore the\neffectiveness of the automatic PRM with both verification and reinforcement learning scenarios.\n3\nMETHODOLOGY\nIn this section, we first present our task formulation to evaluate the performance of reward models\n(§3.1). Subsequently, we outline two typical categories of reward models, ORM and PRM(§3.2).\nThen, we introduce our methodology to automatically build the training dataset for PRM(§3.3),\nbreaking the bottleneck of heavy reliance on manual annotation in existing work (Uesato et al., 2022;\nLightman et al., 2023).\n3.1\nTASK FORMULATION\nWe evaluate the performance of the reward model in two scenarios:\nVerification\nFollowing (Lightman et al., 2023), we consider a best-of-N selection evaluation\nparadigm. Specifically, given a problem p in the testing set, we sample N candidate solutions from a\ngenerator. These candidates are then scored using a reward model, and the highest-scoring solution\nis selected as the final answer. An enhanced reward model elevates the likelihood of selecting the\nsolution containing the correct answer, consequently raising the success rate in solving mathematical\nproblems for LLMs.\n\n\nReinforcement learning\nWe also use the automatically constructed PRM to supervise LLMs with\nstep-by-step PPO. In this scenario, we evaluate the accuracy of the LLMs’ greedy decoding output.\nAn enhanced reward model is instrumental in training higher-performing LLMs.\n3.2\nREWARD MODELS FOR MATHEMATICAL PROBLEM\nORM\nGiven a mathematical problem p and its solution s, ORM (P × S →R) assigns a single\nreal-value to s to indicate whether s is correct. ORM is usually trained with a cross-entropy loss\n(Cobbe et al., 2021; Li et al., 2023b):\nLORM = ys log rs + (1 −ys) log(1 −rs),\n(1)\nwhere ys is the golden answer of the solution s, ys = 1 if s is correct, otherwise ys = 0. rs is\nthe sigmoid score of s assigned by ORM. The success of the reward model hinges on the effective\nconstruction of the high-quality training dataset. As the math problem usually has a certain answer,\nwe can automatically construct the training set of ORM by two steps: 1) sampling some candidate\nsolutions for a problem from a generator; 2) assigning the label to each sampling solution by checking\nwhether its answer is correct. Although false positives solutions that reach the correct answer with\nincorrect reasoning will be misgraded, previous studies have proven that it is still effective for training\na good ORM (Lightman et al., 2023; Yu et al., 2023a).\nPRM\nTake a step further, PRM (P × S →R+) assigns a score to each reasoning step of s, which\nis usually trained with:\nLP RM =\nK\nX\ni=1\nysi log rsi + (1 −ysi) log(1 −rsi),\n(2)\nwhere ysi is the golden answer of si (the i-th step of s), rsi is the sigmoid score of si assigned by\nPRM and K is the number of reasoning steps for s. (Lightman et al., 2023) also conceptualizes the\nPRM training as a three-class classification problem, in which each step is classified as either ‘good’,\n‘neutral’, or ‘bad’. In this paper, we found that there is not much difference between the binary\nand the three classifications, and we regard PRM training as the binary classification. Compared\nto ORM, PRM can provide more detailed and reliable feedback (Lightman et al., 2023). However,\nthere are currently no automated methods available for constructing high-quality PRM training\ndatasets. Previous works (Uesato et al., 2022; Lightman et al., 2023) typically resort to costly human\nannotations. While PRM manages to outperform ORM (Lightman et al., 2023), the annotation cost\ninvariably impedes both the development and application of PRM.\n3.3\nAUTOMATIC PROCESS ANNOTATION\nIn this section, we propose an automatic process annotation framework to mitigate the annotation\ncost issues associated with PRM. We first define the quality of a reasoning step, followed by the\nintroduction of our solution that obviates the necessity for human annotation.\n3.3.1\nDEFINITION\nInspired by Monto Carlo Tree Search (Kocsis & Szepesv´\nari, 2006; Coulom, 2006; Silver et al., 2016;\n´\nSwiechowski et al., 2023), we define the quality of a reasoning step as its potential to deduce the\ncorrect answer. This criterion stems from the primary objective of the reasoning process, which\nessentially is a cognitive procedure aiding humans or intelligent agents in reaching a well-founded\noutcome (Huang & Chang, 2023). Therefore, a step that has the potential to deduce a well-founded\nresult can be considered a good reasoning step. Analogous to ORM, this definition also introduces\nsome degree of noise. Nevertheless, we find that it is beneficial for effectively training a good PRM.\n3.3.2\nSOLUTION\nCompletion\nTo quantify and estimate the potential for a give reasoning step si, as shown in\nFigure 2, we use a ‘completer’ to finalize N subsequent reasoning processes from this step:\n{(si+1,j, · · · , sKj,j, aj)}N\nj=1, where aj and Kj are the decoded answer and the total number of\nsteps for the j-th finalized solution, respectively. Then, we estimate the potential of this step based on\nthe correctness of all decoded answers A = {aj}N\nj=1.\n\n\nProblem: Let 𝑝𝑝(𝑥𝑥) be a monic polynomial of degree 4.  Three \nof the roots of p(x) are 1, 2, and 3.  Find p(0) + p(4).\nGolden Answer: 24\nAnswer: 20\nSolution: 𝑺𝑺= 𝒔𝒔𝟏𝟏, 𝒔𝒔𝟐𝟐, 𝒔𝒔𝟑𝟑, ⋯, 𝒔𝒔𝑲𝑲 \n(a) Outcome Annotation: 𝒚𝒚𝑺𝑺= 𝟎𝟎\nProblem: …. \n𝒔𝒔𝟏𝟏: Since three of the \nroots of p(x) are 1, 2, and \n3, we can write : p(x) = \n(x - 1)(x - 2)(x - 3)(x - r).\n𝒔𝒔𝟐𝟐,𝟏𝟏\n𝒔𝒔𝟑𝟑,𝟏𝟏\n𝒔𝒔𝑲𝑲𝟏𝟏,𝟏𝟏\n⋯\n𝒔𝒔𝟐𝟐,𝟐𝟐\n𝒔𝒔𝟐𝟐,𝟐𝟐\n𝒔𝒔𝑲𝑲𝟐𝟐,𝟐𝟐\n⋯\n𝒔𝒔𝟐𝟐,𝟑𝟑\n𝒔𝒔𝟐𝟐,𝟑𝟑\n𝒔𝒔𝑲𝑲𝟑𝟑,𝟑𝟑\n⋯\nAnswer: 24\nAnswer: 24\nAnswer: 20\n(b): Process Annotation: 𝒚𝒚𝒔𝒔𝟏𝟏\n𝑺𝑺𝑺𝑺= 𝟐𝟐\n𝟑𝟑 ; 𝒚𝒚𝒔𝒔𝟏𝟏\n𝑯𝑯𝑯𝑯=  𝟏𝟏\n𝒔𝒔𝒊𝒊,𝒋𝒋: the i-th step of the j-th finalized solution.\n𝒔𝒔𝒊𝒊: the i-th step of the solution 𝑺𝑺. \n✓\n✓\n✗\n✗\nFigure 2: Comparison for previous automatic outcome annotation and our automatic process an-\nnotation. (a): automatic outcome annotation assigns a label to the entire solution S, dependent on\nthe correctness of the answer; (b) automatic process annotation employs a ‘completer’ to finalize N\nreasoning processes (N=3 in this figure) for an intermediate step (s1 in this figure), subsequently use\nhard estimation (HE) and soft estimation (SE) to annotate this step based on all decoded answers.\nEstimation\nIn this paper, we use two methods to estimate the quality ysi for the step si, hard\nestimation (HE) and soft estimation (SE). HE supposes that a reasoning step is good as long as it can\nreach the correct answer a∗:\nyHE\nsi\n=\n\u001a1\n∃aj ∈A, aj = a∗\n0\nOtherwise\n(3)\nSE assumes the quality of a step as the frequency with which it reaches the correct answer:\nySE\nsi\n=\nPN\nj=1 I(aj = a∗)\nN\n.\n(4)\nOnce we gather the label of each step, we can train PRM with the cross-entropy loss. In conclusion,\nour automatic process annotation framework defines the quality of a step as its potential to deduce\nthe correct answer and achieve the label of each step by completion and estimation.\n3.4\nRANKING FOR VERIFICATION\nFollowing (Lightman et al., 2023), we use the minimum score across all steps to represent the final\nscore of a solution assigned by PRM. We also explore the combination of self-consistency and reward\nmodels following (Li et al., 2023b). In this context, we initially classify solutions into distinct groups\naccording to their final answers. Following that, we compute the aggregate score for each group.\nFormally, the final prediction answer based on N candidate solutions is:\nasc+rm = arg max\na\nN\nX\ni=1\nI(ai = a) · RM(p, Si).\n(5)\nWhere RM(p, Si) is the score of the i-th solution assigned by ORM or PRM for problem p.\n3.5\nREINFORCE LEARNING WITH PROCESS SUPERVISION\nUpon achieving PRM, we employ reinforcement learning to train LLMs. We implement Proximal\nPolicy Optimization (PPO) in a step-by-step manner. This method differs from the conventional\nstrategy that utilizes PPO with ORM, which only offers a reward at the end of the response. Conversely,\nour step-by-step PPO offers rewards at the end of each reasoning step.\n\n\nModels\nVerifiers\nGSM8K\nMATH500\nLLaMA2-70B: MetaMATH\nSelf-Consistency\n88.0\n39.4\nORM\n91.8\n40.4\nSelf-Consistency + ORM\n92.0\n42.0\nMATH-SHEPHERD (Ours)\n93.2\n44.5\nSelf-Consistency + MATH-SHEPHERD (Ours)\n92.4\n45.2\nLLemma-34B: MetaMATH\nSelf-Consistency\n82.6\n44.2\nORM\n90.0\n43.7\nSelf-Consistency + ORM\n89.6\n45.4\nMATH-SHEPHERD (Ours)\n90.9\n46.0\nSelf-Consistency + MATH-SHEPHERD (Ours)\n89.7\n47.3\nDeepSeek-67B: MetaMATH\nSelf-Consistency\n88.2\n45.4\nORM\n92.6\n45.3\nSelf-Consistency + ORM\n92.4\n47.0\nMATH-SHEPHERD (Ours)\n93.3\n47.0\nSelf-Consistency + MATH-SHEPHERD (Ours)\n92.5\n48.1\nTable 1: Performances of different LLMs on GSM8K and MATH with different verification strategies.\nThe reward models are trained based on LLama2-70B and LLemma-34B on GSM8K and MATH,\nrespectively. The verification is based on 256 outputs.\n4\nEXPERIMENTS\nDatasets\nWe conduct our experiments using two widely used math reasoning datasets, GSM8K\n(Cobbe et al., 2021) and MATH (Hendrycks et al., 2021). For the GSM8K dataset, we leverage the\nwhole test set in both verification and reinforcement learning scenarios. For the MATH dataset, in the\nverification scenario, due to the computation cost, we employ a subset MATH500 that is identical to\nthe test set of Lightman et al. (2023). The subset consists of 500 representative problems, and we\nfind that the subset evaluation produces similar results to the full-set evaluation. To assess different\nverification methods, we generate 256 candidate solutions for each test problem. We report the mean\naccuracy of 3 groups of sampling results. In the reinforcement learning scenario, we use the whole\ntest set to evaluate the model performance. We train LLMs with MetaMATH (Yu et al., 2023b).\nParameter Setting\nOur experiments are based on a series of large language models, LLaMA2-\n7B/13B/70B (Touvron et al., 2023), LLemma-7B/34B (Azerbayev et al., 2023), Mistral-7B (Jiang\net al., 2023) and DeepSeek-67B (DeepSeek, 2023). We train the generator and completer for 3 epochs\non MetaMATH. We train the Mistral-7B with a learning rate of 5e-6. For other models, The learning\nrates are set to 2e-5, 1e-5, and 6e-6 for the 7B/13B, 34B, and 67B/70B LLMs, respectively. To\nconstruct the training dataset of ORM and PRM, we train 7B and 13B models for a single epoch on\nthe GSM8K and MATH training sets. Subsequently, we sample 15 solutions per problem from each\nmodel for the training set. Following this, we eliminate duplicate solutions and annotate the solutions\nat each step. We use LLemma-7B as the completer with the decoded number N=8. Consequently,\nwe obtain around 170k solutions for GSM8K and 270k solutions for MATH. For verification, we\nchoose LLaMA2-70B and LLemma-34B as the base models to train reward models for GSM8K and\nMATH, respectively. For reinforcement learning, we choose Mistral-7B as the base model to train\nreward models and use it to supervise LLama2-7B and Mistral-7B generators. The reward model is\ntrained in 1 epoch with a learning rate 1e-6. For the sake of convenience, we train the PRM using\nthe hard estimation version because it allows us to utilize a standard language modeling pipeline by\nselecting two special tokens to represent ‘has potential’ and ‘no potential’ labels, thereby eliminating\nthe need for any specific model adjustments. In reinforcement learning, the learning rate is 4e-7 and\n1e-7 for LLaMA2-7B and Mistral-7B, respectively. The Kullback-Leibler coefficient is set to 0.04.\nWe implement a cosine learning rate scheduler, employing a minimal learning rate set to 1e-8. We\nuse 3D parallelism provided by hfai1 to train all models with the max sequence length of 512.\n1https://doc.hfai.high-flyer.cn/index.html\n\n\nModels\nGSM8K\nMATH\nLLaMA2-7B: MetaMATH\n66.6\n19.2\n+ RFT\n68.5\n19.9\n+ ORM-PPO\n70.8\n20.8\n+ MATH-SHEPHERD-step-by-step-PPO (Ours)\n73.2\n21.6\nMistral-7B: MetaMATH\n77.9\n28.6\n+ RFT\n79.0\n29.9\n+ ORM-PPO\n81.8\n31.3\n+ MATH-SHEPHERD-step-by-step-PPO (Ours)\n84.1\n33.0\nTable 2: Performances of different 7B models on GSM8K and MATH with greedy decoding. We\nuse the questions in MetaMATH for RFT and PPO training. Both LLaMA2-7B and Mistral-7B are\nsupervised by Mistral-7B-ORM and -MATH-SHEPHERD.\nBaselines and Metrics\nIn the verification scenario, following (Lightman et al., 2023), we evaluate\nthe performance of our reward model by comparing it against the Self-consistency (majority voting)\nand outcome reward model. The accuracy of the best-of-N solution is utilized as the evaluation\nmetric. For PRM, the minimum score across all steps is adopted to represent the final score of a\nsolution. In the reinforcement scenario, we compare our step-by-step supervision with the outcome\nsupervision provided by ORM, and Rejective Sampling Fine-tuning (RFT) (Yuan et al., 2023), we\nsample 8 responses for each question in MetaMATH for RFT. We use the accuracy of LLMs’ greedy\ndecoding output to assess the performance.\n4.1\nMAIN RESULTS\nMATH-SHEPHERD as verifier\nTable 1 presents the performance comparison of various methods\non GSM8K and MATH. We find that: 1) As the verifier, MATH-SHEPHERD consistently outperforms\nself-consistency and ORM on two datasets with all generators. Specifically, enhanced by MATH-\nSHEPHERD, DeepSeek-67B achieves 93.3% and 48.1% accuracy on GSM8K and MATH; 2) In\ncomparison to GSM8K, PRM achieves a greater advantage over ORM on the more challenging\nMATH dataset; This outcome aligns with the findings in Uesato et al. (2022) and Lightman et al.\n(2023). The former discovers that PRM and ORM yield similar results on GSM8K, whereas the latter\nshows that PRM significantly outperforms ORM on the MATH dataset. This could be attributed to the\nrelative simplicity of the GSM8K dataset compared to MATH, i.e., the GSM8K dataset necessitates\nfewer steps for problem-solving. As a result, ORM operates efficiently when handling this particular\ndataset; 3) In GSM8K, when combined with self-consistency, there’s a drop in performance, whereas\nin MATH, performance improves. These results indicate that if the reward model is sufficiently\npowerful for a task, combining it with self-consistency may harm the verification performance.\nMATH-SHEPHERD as reward model on reinforcement learning\nTable 2 presents the performance\nof different LLMs with greedy decoding outputs. As is shown: 1) step-by-step PPO significantly\nimproves the performance of two supervised fine-tuned models. For example, Mistral-7B with\nstep-by-step PPO achieves 84.1% and 33.0% on the GSM8K and MATH datasets, respectively; 2)\nRFT only slightly improves the model performance, we believe this is because MetaMATH already\nhas conducted some data augmentation strategies like RFT; 3) the vanilla PPO with ORM can also\nenhance the model performance. However, it does not perform as well as the step-by-step PPO\nsupervised by MATH-SHEPHERD, demonstrating the potential of step-by-step supervision.\nMATH-SHEPHERD as both reward models and verifiers\nWe also combine the reinforcement\nlearning and the verification. As shown in Table 3: 1) reinforcement learning and verification are\ncomplementary. For example, in MATH, step-by-step PPO Mistral-7B outperforms supervised\nfine-tuning Mistral-7B 7.2% accuracy with self-consistency as the verifier; The performance gap\nis even larger than that of greedy decoding results, i.e., 4.4%; 2) after reinforcement learning, the\nvanilla verification methods with only reward models is inferior to self-consistency, we think the\n\n\nModels\nVerifiers\nGSM8K\nMATH500\nMistral-7B: MetaMATH\nSelf-Consistency\n83.9\n35.1\nORM\n86.2\n36.4\nSelf-Consistency + ORM\n86.6\n38.0\nMATH-SHEPHERD (Ours)\n87.1\n37.3\nSelf-Consistency + MATH-SHEPHERD (Ours)\n86.3\n38.3\nMistral-7B: MetaMATH\nSelf-Consistency\n87.4\n42.3\nORM\n87.6\n41.3\n+step-by-step PPO (Ours)\nSelf-Consistency + ORM\n89.0\n43.1\nMATH-SHEPHERD (Ours)\n88.4\n41.1\nSelf-Consistency + MATH-SHEPHERD (Ours)\n89.1\n43.5\nTable 3: Results of reinforcement learning and verification combination. The reward models are\ntrained based on Mistral-7B. The verification is based on 256 outputs.\nreason is that the initial reward model is not sufficient to supervise the more powerful model after\nPPO. These results can also show the potential of iterative reinforcement learning, which we leave\nfor future work.\n5\nANALYSIS\n5.1\nPERFORMANCE WITH DIFFERENT NUMBER OF CANDIDATE SOLUTIONS\nFigure 3 illustrates the performance comparison of various strategies when applied to different\nnumbers of candidates ranging from 1 to 256 on two benchmarks. The key observations are as\nfollows: 1) PRM exhibits consistent superior performance when compared to both ORM and majority\nvoting, with the degree of this superiority becoming more pronounced as N escalates. 2) In MATH, our\nautomatically annotated datasets outperform the human-annotated PRM800K (Lightman et al., 2023).\nWe ascribe this superiority to the distribution gap and the data quantity. Specifically, PRM800K is\nannotated based on the output from GPT-4, and consequently, a discrepancy arises for the output of\nopen-source LLaMA models fine-tuned on MetaMATH. Furthermore, when considering the quantity\nof data, our automated reward model data exhibits both high scalability and a reduced labeling cost.\nConsequently, our dataset is four times larger than that provided in PRM800K. Overall, these results\nfurther underscore the effectiveness and potential of our method.\n5.2\nQUALITY OF THE AUTOMATIC PROCESS ANNOTATIONS\nIn this section, we explore the quality of our automatic PRM dataset. To achieve this, we manually\nannotate 160 steps sampled from the training set of GSM8K and use different completers to infer\nfrom each step to achieve their label. We find that:\nAutomatic process annotation exhibits satisfactory quality.\nFigure 4(a) demonstrates that uti-\nlizing LLaMA2-70B trained on MetaMATH as the completer, the accuracy of the hard estimation\n(HE) reaches 86% when N equals 4. This suggests that our automatically constructed dataset is of\nhigh quality. However, we observed a decline in the accuracy of the constructed dataset with further\nincreases in N. Our analysis indicates that larger values for N may lead to false positives.\nFigure 4(b) shows the cross-entropy loss between SE and HE labels compared to the human-annotated\ndistribution: as N increases, SE progressively aligns closer to the standard distribution, in contrast\nto HE which does not exhibit similar behavior. It is essential to note that at N=4, HE achieves an\naccuracy of 86%. We can theoretically attain higher quality data exceeding 86% accuracy by utilizing\nSE. However, we discovered that the performance of the verifier exhibits no substantial divergence\nwhether trained with either SE or HE. This may be attributable to the already high-quality annotations\nprovided by HE.\nFurthermore, we also delve into other automatic process annotation methodologies. For instance, (Li\net al., 2023b) employs a natural language inference (NLI) model and a string match rule to annotate a\n\n\n1\n4\n16\n64\n256\nN = number of solutions per problem\n80.0\n82.5\n85.0\n87.5\n90.0\n92.5\n% Problems Soluved (Best-of-N)\nGSM8K\nSC\nORM\nSHEPHERD\n1\n4\n16\n64\n256\nN = number of solutions per problem\n30\n35\n40\n45\n% Problems Soluved (Best-of-N)\nMATH\nSC\nORM\nPRM800K\nSHEPHERD\nFigure 3: Performance of LLaMA2-70B using different verification strategies across different\nnumbers of solution candidates on GSM8K and MATH.\n1\n4\n16\n64\n256\nN = number of decoded path\n80\n82\n84\n86\n% Accuracy\n7B\n13B\n70B\n1\n4\n16\n64\n256\nN = number of decoded path\n1.0\n1.5\n2.0\n2.5\n3.0\nLoss\n7B:Soft\n13B:Soft\n70B:Soft\n70B:Hard\n1\n4\n16\n64\n256\nN = number of decoded path\n1\n2\n3\n4\nLoss\nNormal\nWeak\nAugmented\nFigure 4: Quality of process annotation on GSM8K. (a): Accuracy of the process annotation using\ndifferent completer; (b): Loss of the process annotation using different completer; (c): Loss of the\nprocess annotation using the same completer with different training data.\ngiven step. The NLI-based method annotates a step as correct if it is entailment with any step in the\nreference solutions. The Rule-based method annotates a step as correct if its support number precisely\nmatches that of any steps in the reference solutions. As demonstrated in Table 4, our annotation\nstrategy exhibits substantial superiority over the two approaches.\nThe ability of the LLM completer plays an important role in the data quality.\nWe employ\na completer to finalize multiple subsequent reasoning processes for a given step. Therefore, we\ninvestigate the impact of the LLM completer.\nFigure 4(b) presents the cross-entropy loss across diverse completers trained on MetaMath. The\nresults indicate that a larger completer is adept at generating superior-quality datasets. Figure 4(c)\ndepicts the cross-entropy loss of LLaMA2-70B trained with different datasets. ‘Normal’ denotes\nthe original GSM8K training dataset; ‘Weak’ refers to the Normal set excluding examples whose\nquestions are in our 160 evaluation set; while ‘Augmented’ symbolizes MetaMath, an augmented\nversion of the Normal set.\nThe findings suggest that high-quality training sets allow the model to operate more proficiently as a\ncompleter. Importantly, the ‘Weak’ set exhibits a markedly larger loss than other datasets. This insight\ndrives us to infer that LLMs should acquire the questions in advance to enhance their performance\nas completers. We can also conjecture that a stronger foundational model, coupled with superior\ntraining data, could further enhance the quality of automatic annotation.\n5.3\nINFLUENCE OF THE PRE-TRAINED BASE MODELS\nTo conduct an exhaustive evaluation of MATH-SHEPHERD’s effectiveness, we performed a diverse\nrange of experiments using model sizes 7B, 13B, and 70B.\n\n\nMethods\nModels\nAccuracy (%)\nLoss\nDIVERSE-NLI (Li et al., 2023b)\nDeBERTa (He et al., 2020)\n61.3\n5.43\nDIVERSE-NLI (Li et al., 2023b)\nLLaMA2-13B\n75.6\n3.27\nDIVERSE-Rule (Li et al., 2023b)\n-\n75.0\n3.43\nMATH-SHEPHERD\nLLaMA2-13B (N = 4)\n85.0\n2.05\nTable 4: The comparison between NLI/Rule-based automatic process annotation methods from Li\net al. (2023b) and our method.\n1\n4\n16\n64\n256\n(a) N = number of solutions per problem\n62\n64\n66\n68\n70\n72\n74\n% Problems Soluved (Best-of-N)\nGenerator:7B; Verifier:7B\nSC\nORM\nSHEPHERD\n1\n4\n16\n64\n256\n(b) N = number of solutions per problem\n68\n70\n72\n74\n76\n78\n80\nGenerator:13B; Verifier:13B\nSC\nORM\nSHEPHERD\n1\n4\n16\n64\n256\n(c) N = number of solutions per problem\n81\n82\n83\n84\n85\n86\n87\n88\nGenerator:70B; Verifier:7B\nSC\nORM\nSHEPHERD\n1\n4\n16\n64\n256\n(d) N = number of solutions per problem\n65\n70\n75\n80\n85\nGenerator:7B; Verifier:70B\nSC\nORM\nSHEPHERD\nFigure 5: Performance of different verification strategies on different sizes of generators and verifiers.\nFigures 5(a), 5(b), and 3(a) display the results from the 7B, 13B, and 70B generators paired with\nequal-sized reward models, respectively. It becomes evident that PRM exhibits superiority over\nself-consistency and ORM across all sizes of base models. Moreover, bigger reward models prove\nto be more robust; for instance, the accuracy of the 70B reward models escalates as the number of\ncandidate solutions rises, while the 7B reward models show a decreasing trend.\nFigure 5(c) and 5(d) presents the performance of 7B and 70B generators interfaced with different-\nsized reward models. The findings illustrate that utilizing a larger reward model to validate the output\nof a smaller generator significantly enhances performance. Conversely, when a smaller reward model\nis employed to validate the output of a larger generator, the verification process adversely impacts the\nmodel’s performance compared to SC. These results substantiate that we should utilize a more potent\nreward model for validating or supervising the generator.\n5.4\nINFLUENCE OF THE NUMBER OF DATA\nWe delve deeper into the analysis of PRM and ORM by utilizing varying quantities of training data.\nAs depicted in Figure 6(a), it is clear that PRM exhibits superior data efficiency. Specifically, it\noutperforms ORM by approximately 4% accuracy when applying a modestly sized training dataset\n(i.e., 10k instances). Furthermore, PRM seems to have a higher potential ceiling than ORM. These\nobservations highlight the efficacy of PRM for verification purposes.\n5.5\nOUT-OF-DISTRIBUTION PERFORMANCE\nTo further demonstrate the effectiveness of our method, we conduct an out-of-distribution evaluation\non the Hungarian national final exam2, which consists of 33 questions. The total score of these\nquestions is 100. We use the LLemma-34B trained on MetaMATH to serve as the generator and\ngenerate 256 candidate solutions for each question. We use LLemma-34B-ORM and LLemma-\n34B-PRM to select the solution for each question. As shown in Figure 6(b): 1) both LLemma-\n34B-ORM and LLemma-34B-PRM outperform the origin LLemma-34B, showing the reward model\ncan generalize to other domains; 2) PRM outperforms ORM 9 scores, further demonstrating the\nsuperiority of PRM.\n2https://huggingface.co/datasets/keirp/hungarian_national_hs_finals_\nexam\n\n\n10k\n20k\n40k\n80k\n160k\nNumber of training solutions\n88\n90\n92\n% Problems Soluved (Best-of-256)\nSC\nORM\nSHEPHERD\nGreedy\nORM\nSHEPHERD\nMethods\n30\n40\n50\n60\n70\nScore\n46.0\n54.0\n63.0\nFigure 6: (a): Performance of different reward models using different numbers of training data; (b)\nperformance of different verification strategies on the out-of-distribution Hungarian national exam.\nQuestion\nThe first term of an arithmetic sequence is 18. The sum of the first six terms of the sequence is equal to the sum of the first\nseven terms. Show that the sum of the first 13 terms of the sequence is 0 and calculate the 13th term of the sequence.\nMATH-SHEPHERD\nStep 1: Let the common difference of the arithmetic sequence be d. MATH-SHEPHERD: 0.99.\nStep 2: The sum of the first six terms of the sequence is 18 + (18 + d) + (18 + 2d) + (18 + 3d) + (18 + 4d) +\n(18 + 5d) = 108 + 30d. MATH-SHEPHERD: 0.85.\n......\nStep 6: Therefore, d = −3. MATH-SHEPHERD: 0.85.\n......\nStep 8: The 13th term of the sequence is 18 −36 = −18. MATH-SHEPHERD: 0.93. ORM Score: 0.73.\nORM\nStep 1: Let the common difference of the arithmetic sequence be d. MATH-SHEPHERD: 0.99.\nStep 2: The sum of the first six terms is 18 + (18 + d) + (18 + 2d) + (18 + 3d) + (18 + 4d) + (18 + 5d) =\n90 + 15d. MATH-SHEPHERD: 0.53.\n......\nStep 6: Dividing by −6, we find that d = −2. MATH-SHEPHERD: 0.38.\n......\nStep 9: The 13th term of the sequence is 18 −26 = −8. MATH-SHEPHERD: 0.38. ORM Score: 0.84.\nTable 5: A case study from the Hungarian national exam. Red text denotes the mistake that ORM\nfails to detect.\nWe also conduct a case study to intuitively demonstrate the effectiveness of MATH-SHEPHERD. As\noutlined in Table 5, when presented with a question from the Hungarian national final exam, our\nMATH-SHEPHERD accurately selected the correct solution from a pool of 256 potential solutions,\nwhich ORM failed. Moreover, MATH-SHEPHERD displayed superior discernment by precisely\nidentifying incorrect steps within the solutions selected by ORM. Notably, it recognized errors in\nStep 2, Step 6, and Step 9 and so on, and subsequently assigned them lower scores relative to those\nfor steps present in the correct solutions.\n6\nLIMITATIONS\nOur paper has some limitations, which we leave for future work:\nThe computational cost of the completion process.\nTo determine the label of each reasoning\nstep, we utilize a ‘completer’ to decode N subsequent reasoning processes. We observe that as N\nincreases, so does the quality of automatic annotations. However, this completion process demands a\nlot of computing resources, potentially imposing a limitation on the usage of our method. Despite\nthis limitation, the cost remains significantly lower than human annotation. Furthermore, we are\noptimistic that advancements in efficient inference techniques such as speculative decoding (Xia\net al., 2022; Leviathan et al., 2023) and vLLM (Kwon et al., 2023) could mitigate this limitation.\nThe automatic process annotation consists of noise.\nSimilar to the automatic outcome annotation,\nour automatic process annotation also has noise. Despite this, our experiments verify the efficacy\nof our method for training a PRM. In particular, the PRM trained on our dataset outperforms the\n\n\nhuman-annotated PRM800K dataset. However, a noticeable gap remains between PRM800K and the\ncandidate responses generated by the open-source models utilized in this study, which may result in\nthe invalidation of PRM800K. As a result, the impact of this potential noise on PRM performance\nis still undetermined. A comprehensive comparison between human and automated annotations is\nenvisaged for future studies. Furthermore, we assert that integrating human and automated process\nannotations could play a vital role in constructing robust and efficient process supervision.\n7\nCONCLUSION\nIn this paper, we introduce a process-oriented math verifier called MATH-SHEPHERD, which assigns a\nreward score to each step of the LLM’s outputs on math problems. The training of MATH-SHEPHERD\nis achieved using automatically constructed process-wise supervision data, thereby eradicating the\nnecessity for labor-intensive human annotation. Remarkably, this automatic methodology correlates\nstrongly with human annotations. Extensive experiments in both verification and reinforcement\nlearning scenarios demonstrate the effectiveness of our method.\nREFERENCES\nRohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos,\nSiamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm 2 technical report. arXiv\npreprint arXiv:2305.10403, 2023.\nZhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen McAleer, Albert Q\nJiang, Jia Deng, Stella Biderman, and Sean Welleck. Llemma: An open language model for\nmathematics. arXiv preprint arXiv:2310.10631, 2023.\nZhen Bi, Ningyu Zhang, Yinuo Jiang, Shumin Deng, Guozhou Zheng, and Huajun Chen. When do\nprogram-of-thoughts work for reasoning? arXiv preprint arXiv:2308.15452, 2023.\nS´\nebastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar,\nPeter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence:\nEarly experiments with gpt-4. arXiv preprint arXiv:2303.12712, 2023.\nLiang Chen, Yichi Zhang, Shuhuai Ren, Haozhe Zhao, Zefan Cai, Yuchi Wang, Peiyi Wang, Tianyu\nLiu, and Baobao Chang. Towards end-to-end embodied decision making via multi-modal large\nlanguage model: Explorations with gpt4-vision and beyond. arXiv preprint arXiv:2310.02071,\n2023.\nKarl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser,\nMatthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve\nmath word problems. arXiv preprint arXiv:2110.14168, 2021.\nR´\nemi Coulom. Efficient selectivity and backup operators in monte-carlo tree search. In International\nconference on computers and games, pp. 72–83. Springer, 2006.\nDeepSeek. Deepseek llm: Let there be answers. https://github.com/deepseek-ai/\nDeepSeek-LLM, 2023.\nYao Fu, Hao Peng, Ashish Sabharwal, Peter Clark, and Tushar Khot. Complexity-based prompting\nfor multi-step reasoning. arXiv preprint arXiv:2210.00720, 2022.\nZhibin Gou, Zhihong Shao, Yeyun Gong, Yujiu Yang, Minlie Huang, Nan Duan, Weizhu Chen,\net al. Tora: A tool-integrated reasoning agent for mathematical problem solving. arXiv preprint\narXiv:2309.17452, 2023.\nPengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: Decoding-enhanced bert\nwith disentangled attention. arXiv preprint arXiv:2006.03654, 2020.\nDan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song,\nand Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv\npreprint arXiv:2103.03874, 2021.\n\n\nJie Huang and Kevin Chen-Chuan Chang. Towards reasoning in large language models: A sur-\nvey. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Findings of the As-\nsociation for Computational Linguistics: ACL 2023, pp. 1049–1065, Toronto, Canada, July\n2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.findings-acl.67. URL\nhttps://aclanthology.org/2023.findings-acl.67.\nJie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song,\nand Denny Zhou. Large language models cannot self-correct reasoning yet. arXiv preprint\narXiv:2310.01798, 2023.\nAlbert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot,\nDiego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al.\nMistral 7b. arXiv preprint arXiv:2310.06825, 2023.\nJean Kaddour, Joshua Harris, Maximilian Mozes, Herbie Bradley, Roberta Raileanu, and Robert\nMcHardy. Challenges and applications of large language models. arXiv preprint arXiv:2307.10169,\n2023.\nLevente Kocsis and Csaba Szepesv´\nari. Bandit based monte-carlo planning. In European conference\non machine learning, pp. 282–293. Springer, 2006.\nWoosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph\nGonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model\nserving with pagedattention.\nIn Proceedings of the 29th Symposium on Operating Systems\nPrinciples, pp. 611–626, 2023.\nYaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative\ndecoding. In International Conference on Machine Learning, pp. 19274–19286. PMLR, 2023.\nLei Li, Yuwei Yin, Shicheng Li, Liang Chen, Peiyi Wang, Shuhuai Ren, Mukai Li, Yazheng Yang,\nJingjing Xu, Xu Sun, et al. M3it: A large-scale dataset towards multi-modal multilingual instruction\ntuning. arXiv preprint arXiv:2306.04387, 2023a.\nYifei Li, Zeqi Lin, Shizhuo Zhang, Qiang Fu, Bei Chen, Jian-Guang Lou, and Weizhu Chen.\nMaking language models better reasoners with step-aware verifier. In Anna Rogers, Jordan Boyd-\nGraber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association\nfor Computational Linguistics (Volume 1: Long Papers), pp. 5315–5333, Toronto, Canada, July\n2023b. Association for Computational Linguistics. doi: 10.18653/v1/2023.acl-long.291. URL\nhttps://aclanthology.org/2023.acl-long.291.\nHunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan\nLeike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. arXiv preprint\narXiv:2305.20050, 2023.\nHaipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng,\nQingwei Lin, Shifeng Chen, and Dongmei Zhang. Wizardmath: Empowering mathematical\nreasoning for large language models via reinforced evol-instruct. arXiv preprint arXiv:2308.09583,\n2023.\nQianli Ma, Haotian Zhou, Tingkai Liu, Jianbo Yuan, Pengfei Liu, Yang You, and Hongxia Yang.\nLet’s reward step by step: Step-level reward model as the navigators for reasoning. arXiv preprint\narXiv:2310.10080, 2023.\nOpenAI. GPT-4 technical report. CoRR, abs/2303.08774, 2023. doi: 10.48550/arXiv.2303.08774.\nURL https://doi.org/10.48550/arXiv.2303.08774.\nSarah Pan, Vladislav Lialin, Sherin Muckatira, and Anna Rumshisky. Let’s reinforce step by step.\narXiv preprint arXiv:2311.05821, 2023.\nJoon Sung Park, Joseph O’Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S\nBernstein. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th\nAnnual ACM Symposium on User Interface Software and Technology, pp. 1–22, 2023.\n\n\nDavid Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche,\nJulian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering\nthe game of go with deep neural networks and tree search. nature, 529(7587):484–489, 2016.\nYifan Song, Weimin Xiong, Dawei Zhu, Cheng Li, Ke Wang, Ye Tian, and Sujian Li. Restgpt: Con-\nnecting large language models with real-world applications via restful apis. corr, abs/2306.06624,\n2023. doi: 10.48550. arXiv preprint arXiv.2306.06624.\nMaciej ´\nSwiechowski, Konrad Godlewski, Bartosz Sawicki, and Jacek Ma´\nndziuk. Monte carlo tree\nsearch: A review of recent modifications and applications. Artificial Intelligence Review, 56(3):\n2497–2562, 2023.\nHugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay\nBashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation\nand fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023.\nJonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia\nCreswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process-and\noutcome-based feedback. arXiv preprint arXiv:2211.14275, 2022.\nGuanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and\nAnima Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv\npreprint arXiv:2305.16291, 2023a.\nPeiyi Wang, Lei Li, Liang Chen, Feifan Song, Binghuai Lin, Yunbo Cao, Tianyu Liu, and Zhifang Sui.\nMaking large language models better reasoners with alignment. arXiv preprint arXiv:2309.02144,\n2023b.\nPeiyi Wang, Lei Li, Liang Chen, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and\nZhifang Sui. Large language models are not fair evaluators. arXiv preprint arXiv:2305.17926,\n2023c.\nXuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha\nChowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language\nmodels. In The Eleventh International Conference on Learning Representations, ICLR 2023,\nKigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023d. URL https://openreview.net/\npdf?id=1PL1NIMMrw.\nJason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V.\nLe, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In\nNeurIPS, 2022. URL http://papers.nips.cc/paper_files/paper/2022/hash/\n9d5609613524ecf4f15af0f7b31abca4-Abstract-Conference.html.\nZeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A Smith,\nMari Ostendorf, and Hannaneh Hajishirzi. Fine-grained human feedback gives better rewards for\nlanguage model training. arXiv preprint arXiv:2306.01693, 2023.\nHeming Xia, Tao Ge, Furu Wei, and Zhifang Sui. Lossless speedup of autoregressive translation with\ngeneralized aggressive decoding. arXiv preprint arXiv:2203.16487, 2022.\nFei Yu, Anningzhe Gao, and Benyou Wang. Outcome-supervised verifiers for planning in mathemati-\ncal reasoning. arXiv preprint arXiv:2311.09724, 2023a.\nLonghui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo\nLi, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for\nlarge language models. arXiv preprint arXiv:2309.12284, 2023b.\nZheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Chuanqi Tan, and Chang Zhou. Scaling\nrelationship on learning mathematical reasoning with large language models. arXiv preprint\narXiv:2308.01825, 2023.\nXiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen.\nMammoth: Building math generalist models through hybrid instruction tuning. arXiv preprint\narXiv:2309.05653, 2023.\n\n\nYifan Zhang, Jingqin Yang, Yang Yuan, and Andrew Chi-Chih Yao. Cumulative reasoning with large\nlanguage models. arXiv preprint arXiv:2308.04371, 2023.\nLianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang,\nZi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and\nchatbot arena. arXiv preprint arXiv:2306.05685, 2023.\nXinyu Zhu, Junjie Wang, Lin Zhang, Yuxiang Zhang, Yongfeng Huang, Ruyi Gan, Jiaxing Zhang,\nand Yujiu Yang. Solving math word problems via cooperative reasoning induced language models.\nIn Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual\nMeeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 4471–4485,\nToronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.\nacl-long.245. URL https://aclanthology.org/2023.acl-long.245.","difficulty":"easy","domain":"Multi-Document QA","length":"short","question":"Compared with lets verify step by step, which of the following points is not included in the improvement of math-shepherd?","sub_domain":"Academic"}

Source: https://huggingface.co/datasets/zai-org/LongBench-v2

initial import

Posting: /agents

GET /api/v1/write?intent=publish&task_id=557c57e4-7d0b-54ed-9e52-736b2b460ce3&body={url_encoded_text}&agent_name={optional_name}&nonce={optional_random_id}