LongBench v2 / 66fab090bb02136c067c74e7 / Regarding the comparison of the methods for improving math ability in these two…

Abstract
Mathematical reasoning poses a significant challenge for language models due to its complex
and structured nature. In this paper, we introduce DeepSeekMath 7B, which continues pre-
training DeepSeek-Coder-Base-v1.5 7B with 120B math-related tokens sourced from Common
Crawl, together with natural language and code data. DeepSeekMath 7B has achieved an
impressive score of 51.7% on the competition-level MATH benchmark without relying on
external toolkits and voting techniques, approaching the performance level of Gemini-Ultra
and GPT-4. Self-consistency over 64 samples from DeepSeekMath 7B achieves 60.9% on MATH.
The mathematical reasoning capability of DeepSeekMath is attributed to two key factors: First,
we harness the significant potential of publicly available web data through a meticulously
engineered data selection pipeline. Second, we introduce Group Relative Policy Optimization
(GRPO), a variant of Proximal Policy Optimization (PPO), that enhances mathematical reasoning
abilities while concurrently optimizing the memory usage of PPO.
Figure 1 | Top1 accuracy of open-source models on the competition-level MATH benchmark
(Hendrycks et al., 2021) without the use of external toolkits and voting techniques.
∗Core contributors.
† Work done during internship at DeepSeek-AI.
arXiv:2402.03300v3  [cs.CL]  27 Apr 2024


1. Introduction
Large language models (LLM) have revolutionized the approach to mathematical reasoning
in artificial intelligence, spurring significant advancements in both the quantitative reasoning
benchmark (Hendrycks et al., 2021) and the geometry reasoning benchmark (Trinh et al., 2024).
Moreover, these models have proven instrumental in assisting humans in solving complex
mathematical problems (Tao, 2023). However, cutting-edge models such as GPT-4 (OpenAI,
2023) and Gemini-Ultra (Anil et al., 2023) are not publicly available, and the currently accessible
open-source models considerably trail behind in performance.
In this study, we introduce DeepSeekMath, a domain-specific language model that signifi-
cantly outperforms the mathematical capabilities of open-source models and approaches the
performance level of GPT-4 on academic benchmarks. To achieve this, we create the DeepSeek-
Math Corpus, a large-scale high-quality pre-training corpus comprising 120B math tokens. This
dataset is extracted from the Common Crawl (CC) using a fastText-based classifier (Joulin et al.,
2016). In the initial iteration, the classifier is trained using instances from OpenWebMath (Paster
et al., 2023) as positive examples, while incorporating a diverse selection of other web pages to
serve as negative examples. Subsequently, we employ the classifier to mine additional positive
instances from the CC, which are further refined through human annotation. The classifier is
then updated with this enhanced dataset to improve its performance. The evaluation results
indicate that the large-scale corpus is of high quality, as our base model DeepSeekMath-Base
7B achieves 64.2% on GSM8K (Cobbe et al., 2021) and 36.2% on the competition-level MATH
dataset (Hendrycks et al., 2021), outperforming Minerva 540B (Lewkowycz et al., 2022a). In
addition, the DeepSeekMath Corpus is multilingual, so we notice an improvement in Chinese
mathematical benchmarks (Wei et al., 2023; Zhong et al., 2023). We believe that our experience
in mathematical data processing is a starting point for the research community, and there is
significant room for improvement in the future.
DeepSeekMath-Base is initialized with DeepSeek-Coder-Base-v1.5 7B (Guo et al., 2024), as
we notice that starting from a code training model is a better choice compared to a general
LLM. Furthermore, we observe the math training also improves model capability on MMLU
(Hendrycks et al., 2020) and BBH benchmarks (Suzgun et al., 2022), indicating it does not only
enhance the model’s mathematical abilities but also amplifies general reasoning capabilities.
After pre-training, we apply mathematical instruction tuning to DeepSeekMath-Base with
chain-of-thought (Wei et al., 2022), program-of-thought (Chen et al., 2022; Gao et al., 2023), and
tool-integrated reasoning (Gou et al., 2023) data. The resulting model DeepSeekMath-Instruct
7B beats all 7B counterparts and is comparable with 70B open-source instruction-tuned models.
Furthermore, we introduce the Group Relative Policy Optimization (GRPO), a variant rein-
forcement learning (RL) algorithm of Proximal Policy Optimization (PPO) (Schulman et al., 2017).
GRPO foregoes the critic model, instead estimating the baseline from group scores, significantly
reducing training resources. By solely using a subset of English instruction tuning data, GRPO
obtains a substantial improvement over the strong DeepSeekMath-Instruct, including both
in-domain (GSM8K: 82.9% →88.2%, MATH: 46.8% →51.7%) and out-of-domain mathematical
tasks (e.g., CMATH: 84.6% →88.8%) during the reinforcement learning phase. We also provide
a unified paradigm to understand different methods, such as Rejection Sampling Fine-Tuning
(RFT) (Yuan et al., 2023a), Direct Preference Optimization (DPO) (Rafailov et al., 2023), PPO and
GRPO. Based on such a unified paradigm, we find that all these methods are conceptualized as
either direct or simplified RL techniques. We also conduct extensive experiments, e.g., online
v.s. offline training, outcome v.s. process supervision, single-turn v.s. iterative RL and so on,
2


to deeply investigate the essential elements of this paradigm. At last, we explain why our RL
boosts the performance of instruction-tuned models, and further summarize potential directions
to achieve more effective RL based on this unified paradigm.
1.1. Contributions
Our contribution includes scalable math pre-training, along with the exploration and analysis of
reinforcement learning.
Math Pre-Training at Scale
• Our research provides compelling evidence that the publicly accessible Common Crawl
data contains valuable information for mathematical purposes. By implementing a metic-
ulously designed data selection pipeline, we successfully construct the DeepSeekMath
Corpus, a high-quality dataset of 120B tokens from web pages filtered for mathemati-
cal content, which is almost 7 times the size of the math web pages used by Minerva
(Lewkowycz et al., 2022a) and 9 times the size of the recently released OpenWebMath
(Paster et al., 2023).
• Our pre-trained base model DeepSeekMath-Base 7B achieves comparable performance
with Minerva 540B (Lewkowycz et al., 2022a), indicating the number of parameters is not
the only key factor in mathematical reasoning capability. A smaller model pre-trained on
high-quality data could achieve strong performance as well.
• We share our findings from math training experiments. Code training prior to math
training improves models’ ability to solve mathematical problems both with and without
tool use. This offers a partial answer to the long-standing question: does code training
improve reasoning abilities? We believe it does, at least for mathematical reasoning.
• Although training on arXiv papers is common, especially in many math-related papers, it
brings no notable improvements on all mathematical benchmarks adopted in this paper.
Exploration and Analysis of Reinforcement Learning
• We introduce Group Relative Policy Optimization (GRPO), an efficient and effective
reinforcement learning algorithm. GRPO foregoes the critic model, instead estimating
the baseline from group scores, significantly reducing training resources compared to
Proximal Policy Optimization (PPO).
• We demonstrate that GRPO significantly enhances the performance of our instruction-
tuned model DeepSeekMath-Instruct, by solely using the instruction-tuning data. Further-
more, we observe enhancements in the out-of-domain performance during the reinforce-
ment learning process.
• We provide a unified paradigm to understand different methods, such as RFT, DPO,
PPO, and GRPO. We also conduct extensive experiments, e.g., online v.s. offline training,
outcome v.s. process supervision, single-turn v.s. iterative reinforcement learning, and so
on to deeply investigate the essential elements of this paradigm.
• Based on our unified paradigm, we explore the reasons behind the effectiveness of rein-
forcement learning, and summarize several potential directions to achieve more effective
reinforcement learning of LLMs.
1.2. Summary of Evaluations and Metrics
• English and Chinese Mathematical Reasoning: We conduct comprehensive assessments
of our models on English and Chinese benchmarks, covering mathematical problems
3


from grade-school level to college level. English benchmarks include GSM8K (Cobbe
et al., 2021), MATH (Hendrycks et al., 2021), SAT (Azerbayev et al., 2023), OCW Courses
(Lewkowycz et al., 2022a), MMLU-STEM (Hendrycks et al., 2020). Chinese benchmarks
include MGSM-zh (Shi et al., 2023), CMATH (Wei et al., 2023), Gaokao-MathCloze (Zhong
et al., 2023), and Gaokao-MathQA (Zhong et al., 2023). We evaluate models’ ability
to generate self-contained text solutions without tool use, and also the ability to solve
problems using Python.
On English benchmarks, DeepSeekMath-Base is competitive with the closed-source Min-
erva 540B (Lewkowycz et al., 2022a), and surpasses all open-source base models (e.g., Mis-
tral 7B (Jiang et al., 2023) and Llemma-34B (Azerbayev et al., 2023)), regardless of whether
they’ve undergone math pre-training or not, often by a significant margin. Notably,
DeepSeekMath-Base is superior on Chinese benchmarks, likely because we don’t follow
previous works (Azerbayev et al., 2023; Lewkowycz et al., 2022a) to collect English-only
math pre-training data, and also include high-quality non-English ones. With mathemati-
cal instruction tuning and reinforcement learning, the resulting DeepSeekMath-Instruct
and DeepSeekMath-RL demonstrate strong performance, obtaining an accuracy of over
50% on the competition-level MATH dataset for the first time within the open-source
community.
• Formal Mathematics: We evaluate DeepSeekMath-Base using the informal-to-formal
theorem proving task from (Jiang et al., 2022) on miniF2F (Zheng et al., 2021) with Isabelle
(Wenzel et al., 2008) chosen to be the proof assistant. DeepSeekMath-Base demonstrates
strong few-shot autoformalization performance.
• Natural Language Understanding, Reasoning, and Code: To build a comprehensive
profile of models’ general understanding, reasoning, and coding capabilities, we eval-
uate DeepSeekMath-Base on the Massive Multitask Language Understanding (MMLU)
benchmark (Hendrycks et al., 2020) which encompasses 57 multiple-choice tasks covering
diverse subjects, BIG-Bench Hard (BBH) (Suzgun et al., 2022) which consists of 23 chal-
lenging tasks that mostly require multi-step reasoning to solve, as well as HumanEval
(Chen et al., 2021) and MBPP (Austin et al., 2021) which are widely used to evaluate code
language models. Math pre-training benefits both language understanding and reasoning
performance.
2. Math Pre-Training
2.1. Data Collection and Decontamination
In this section, we will outline the process of constructing the DeepSeekMath Corpus from
Common Crawl. As depicted in Figure 2, we present an iterative pipeline that demonstrates
how to systematically gather a large-scale mathematical corpus from Common Crawl, starting
with a seed corpus (e.g., a small but high-quality collection of math-related dataset). It’s worth
noting that this approach is also applicable to other domains, such as coding.
First, we choose OpenWebMath (Paster et al., 2023), a collection of high-quality mathematical
web texts, as our initial seed corpus. Using this corpus, we train a fastText model (Joulin et al.,
2016) to recall more OpenWebMath-like mathematical web pages. Specifically, we randomly
select 500,000 data points from the seed corpus as positive training examples and another
500,000 web pages from Common Crawl as negative ones. We employ an open-source library1
for training, configuring the vector dimension to 256, learning rate to 0.1, the maximum length
1https://fasttext.cc
4


Math Seed
Math Corpus
1. Train a FastText Model
2. Recall Math-Related Webpages 
From Common Crawl
3. Discover Math-Related Domains
4. Annotate Math-Related 
URL Path From Labelers
Deduplicated Common Crawl
40B HTML pages
Figure 2 | An iterative pipeline that collects mathematical web pages from Common Crawl.
of word n-gram to 3, the minimum number of word occurrences to 3, and the number of
training epochs to 3. To reduce the size of the original Common Crawl, we employ URL-based
deduplication and near-deduplication techniques, resulting in 40B HTML web pages. We then
recall mathematical web pages from deduplicated Common Crawl with the fastText model.
To filter out low-quality mathematical content, we rank the collected pages according to their
scores predicted by the fastText model, and only preserve the top-ranking ones. The volume
of data preserved is assessed through pre-training experiments on the top 40B, 80B, 120B, and
160B tokens. In the first iteration, we choose to keep the top 40B tokens.
After the first iteration of data collection, numerous mathematical web pages remain un-
collected, mainly because the fastText model is trained on a set of positive examples that lacks
sufficient diversity. We therefore identify additional mathematical web sources to enrich the seed
corpus, so that we can optimize the fastText model. Specifically, we first organize the entire Com-
mon Crawl into disjoint domains; a domain is defined as web pages sharing the same base URL.
For each domain, we calculate the percentage of web pages that are collected in the first iteration.
Domains where over 10% of the web pages have been collected are classified as math-related
(e.g., mathoverflow.net). Subsequently, we manually annotate the URLs associated with
mathematical content within these identified domains (e.g., mathoverflow.net/questions).
Web pages linked to these URLs, yet uncollected, will be added to the seed corpus. This ap-
proach enables us to gather more positive examples, thereby training an improved fastText
model capable of recalling more mathematical data in the subsequent iteration. After four
iterations of data collection, we end up with 35.5M mathematical web pages, totaling 120B
tokens. In the fourth iteration, we notice that nearly 98% of the data has already been collected
in the third iteration, so we decide to cease data collection.
To avoid benchmark contamination, we follow Guo et al. (2024) to filter out web pages
containing questions or answers from English mathematical benchmarks such as GSM8K (Cobbe
et al., 2021) and MATH (Hendrycks et al., 2021) and Chinese benchmarks such as CMATH
(Wei et al., 2023) and AGIEval (Zhong et al., 2023). The filtering criteria are as follows: any
text segment containing a 10-gram string that matches exactly with any sub-string from the
evaluation benchmarks is removed from our math training corpus. For benchmark texts that
are shorter than 10 grams but have at least 3 grams, we employ exact matching to filter out
contaminated web pages.
5


2.2. Validating the Quality of the DeepSeekMath Corpus
We run pre-training experiments to investigate how the DeepSeekMath Corpus is compared
with the recently released math-training corpora:
• MathPile (Wang et al., 2023c): a multi-source corpus (8.9B tokens) aggregated from
textbooks, Wikipedia, ProofWiki, CommonCrawl, StackExchange, and arXiv, with the
majority (over 85%) sourced from arXiv;
• OpenWebMath (Paster et al., 2023): CommonCrawl data filtered for mathematical content,
totaling 13.6B tokens;
• Proof-Pile-2 (Azerbayev et al., 2023): a mathematical corpus consisting of OpenWeb-
Math, AlgebraicStack (10.3B tokens of mathematical code), and arXiv papers (28.0B to-
kens). When experimenting on Proof-Pile-2, we follow Azerbayev et al. (2023) to use an
arXiv:Web:Code ratio of 2:4:1.
2.2.1. Training Setting
We apply math training to a general pre-trained language model with 1.3B parameters, which
shares the same framework as the DeepSeek LLMs (DeepSeek-AI, 2024), denoted as DeepSeek-
LLM 1.3B. We separately train a model on each mathematical corpus for 150B tokens. All
experiments are conducted using the efficient and light-weight HAI-LLM (High-flyer, 2023)
training framework. Following the training practice of DeepSeek LLMs, we use the AdamW
optimizer (Loshchilov and Hutter, 2017) with 𝛽1 = 0.9, 𝛽2 = 0.95, and weight_decay = 0.1, along
with a multi-step learning rate schedule where the learning rate reaches the peak after 2,000
warmup steps, decreases to its 31.6% after 80% of the training process, and further decreases to
10.0% of the peak after 90% of the training process. We set the maximum value of learning rate
to 5.3e-4, and use a batch size of 4M tokens with a 4K context length.
Math Corpus
Size
English Benchmarks
Chinese Benchmarks
GSM8K MATH OCW
SAT
MMLU
STEM
CMATH
Gaokao
MathCloze
Gaokao
MathQA
No Math Training
N/A
2.9%
3.0%
2.9% 15.6%
19.5%
12.3%
0.8%
17.9%
MathPile
8.9B
2.7%
3.3%
2.2% 12.5%
15.7%
1.2%
0.0%
2.8%
OpenWebMath
13.6B
11.5%
8.9%
3.7% 31.3%
29.6%
16.8%
0.0%
14.2%
Proof-Pile-2
51.9B
14.3%
11.2%
3.7% 43.8%
29.2%
19.9%
5.1%
11.7%
DeepSeekMath Corpus 120.2B
23.8%
13.6%
4.8% 56.3%
33.1%
41.5%
5.9%
23.6%
Table 1 | Performance of DeepSeek-LLM 1.3B trained on different mathematical corpora, evalu-
ated using few-shot chain-of-thought prompting. Corpus sizes are calculated using our tokenizer
with a vocabulary size of 100K.
2.2.2. Evaluation Results
The DeepSeekMath Corpus is of high quality, covers multilingual mathematical content, and
is the largest in size.
• High-quality: We evaluate downstream performance on 8 mathematical benchmarks using
few-shot chain-of-thought prompting Wei et al. (2022). As shown in Table 1, there is a clear
performance lead of the model trained on the DeepSeekMath Corpus. Figure 3 shows that
the model trained on the DeepSeekMath Corpus demonstrates better performance than
6


Figure 3 | Benchmark curves of DeepSeek-LLM 1.3B trained on different mathematical corpora.
Proof-Pile-2 at 50B tokens (1 full epoch of Proof-Pile-2), indicating the average quality of
DeepSeekMath Corpus is higher.
• Multilingual: The DeepSeekMath Corpus encompasses data in multiple languages, pre-
dominantly featuring English and Chinese as the two most represented languages. As
shown in Table 1, training on the DeepSeekMath Corpus enhances mathematical reasoning
performance in both English and Chinese. In contrast, existing mathematical corpora,
which are primarily English-centric, show limited improvement and may even hinder
performance in Chinese mathematical reasoning.
• Large-scale: The DeepSeekMath Corpus is several times larger than existing mathematical
corpora. As depicted in Figure 3, DeepSeek-LLM 1.3B, when trained on the DeepSeek-
Math Corpus, shows a steeper learning curve along with more lasting improvements. In
contrast, the baseline corpora are much smaller, and have already been repeated multiple
rounds during training, with the resulting model performance quickly reaching a plateau.
2.3. Training and Evaluating DeepSeekMath-Base 7B
In this section, we introduce DeepSeekMath-Base 7B, a base model with strong reasoning
abilities, especially in mathematics. Our model is initialized with DeepSeek-Coder-Base-v1.5 7B
7


(Guo et al., 2024) and trained for 500B tokens. The distribution of the data is as follows: 56%
is from the DeepSeekMath Corpus, 4% from AlgebraicStack, 10% from arXiv, 20% is Github
code, and the remaining 10% is natural language data from Common Crawl in both English and
Chinese. We mainly adopt the training setting specified in Section 2.2.1, except that we set the
maximum value of the learning rate to 4.2e-4 and use a batch size of 10M tokens.
We conduct a comprehensive assessment of the mathematical capabilities of DeepSeekMath-
Base 7B, focusing on its ability to produce self-contained mathematical solutions without relying
on external tools, solve mathematical problems using tools, and conduct formal theorem proving.
Beyond mathematics, we also provide a more general profile of the base model, including its
performance of natural language understanding, reasoning, and programming skills.
Mathematical Problem Solving with Step-by-Step Reasoning
We evaluate DeepSeekMath-
Base’s performance of solving mathematical problems using few-shot chain-of-thought prompt-
ing (Wei et al., 2022), across eight benchmarks in English and Chinese. These benchmarks encom-
pass quantitative reasoning (e.g., GSM8K (Cobbe et al., 2021), MATH (Hendrycks et al., 2021),
and CMATH (Wei et al., 2023)) and multiple-choice problems (e.g., MMLU-STEM (Hendrycks
et al., 2020) and Gaokao-MathQA (Zhong et al., 2023)), covering diverse fields of mathematics
from elementary to college-level complexity.
As shown in Table 2, DeepSeekMath-Base 7B leads in performance across all eight bench-
marks among the open-source base models (including the widely-used general model Mistral
7B (Jiang et al., 2023) and the recently released Llemma 34B (Azerbayev et al., 2023) which
underwent math training on Proof-Pile-2 (Azerbayev et al., 2023)). Notably, on the competition-
level MATH dataset, DeepSeekMath-Base surpasses existing open-source base models by over
10% absolute, and outperforms Minerva 540B (Lewkowycz et al., 2022a), a closed-source base
model 77 times larger which builds on PaLM (Lewkowycz et al., 2022b) and is further trained
on mathematical texts.
Model
Size
English Benchmarks
Chinese Benchmarks
GSM8K MATH OCW
SAT
MMLU
STEM
CMATH
Gaokao
MathCloze
Gaokao
MathQA
Closed-Source Base Model
Minerva
7B
16.2%
14.1%
7.7%
-
35.6%
-
-
-
Minerva
62B
52.4%
27.6%
12.0%
-
53.9%
-
-
-
Minerva
540B
58.8%
33.6%
17.6%
-
63.9%
-
-
-
Open-Source Base Model
Mistral
7B
40.3%
14.3%
9.2%
71.9%
51.1%
44.9%
5.1%
23.4%
Llemma
7B
37.4%
18.1%
6.3%
59.4% 43.1%
43.4%
11.9%
23.6%
Llemma
34B
54.0%
25.3%
10.3% 71.9%
52.9%
56.1%
11.9%
26.2%
DeepSeekMath-Base 7B
64.2%
36.2% 15.4% 84.4%
56.5%
71.7%
20.3%
35.3%
Table 2 | Comparisons between DeepSeekMath-Base 7B and strong base models on English and
Chinese mathematical benchmarks. Models are evaluated with chain-of-thought prompting.
Minerva results are quoted from Lewkowycz et al. (2022a).
8


Mathematical Problem Solving with Tool Use
We evaluate program-aided mathematical
reasoning on GSM8K and MATH using few-shot program-of-thought prompting (Chen et al.,
2022; Gao et al., 2023). Models are prompted to solve each problem by writing a Python program
where libraries such as math and sympy can be utilized for intricate computations. The execution
result of the program is evaluated as the answer. As shown in Table 3, DeepSeekMath-Base 7B
outperforms the prior state-of-the-art Llemma 34B.
Model
Size
Problem Solving w/ Tools
Informal-to-Formal Proving
GSM8K+Python MATH+Python miniF2F-valid miniF2F-test
Mistral
7B
48.5%
18.2%
18.9%
18.0%
CodeLlama
7B
27.1%
17.2%
16.3%
17.6%
CodeLlama
34B
52.7%
23.5%
18.5%
18.0%
Llemma
7B
41.0%
18.6%
20.6%
22.1%
Llemma
34B
64.6%
26.3%
21.0%
21.3%
DeepSeekMath-Base 7B
66.9%
31.4%
25.8%
24.6%
Table 3 | Few-shot evaluation of base models’ ability to solve mathematical problems using tools
and the ability to conduct informal-to-formal theorem proving in Isabelle.
Formal Mathematics
Formal proof automation is beneficial to ensure the accuracy and relia-
bility of mathematical proofs and enhance efficiency, with increasing attention in recent years.
We evaluate DeepSeekMath-Base 7B on the task of informal-to-formal proving from (Jiang et al.,
2022) which is to generate a formal proof based on an informal statement, a formal counterpart
of the statement, and an informal proof. We evaluate on miniF2F (Zheng et al., 2021), a bench-
mark for formal Olympiad-level mathematics, and generate a formal proof in Isabelle for each
problem with few-shot prompting. Following Jiang et al. (2022), we leverage models to generate
proof sketches, and execute the off-the-shelf automated prover Sledgehammer (Paulson, 2010)
to fill in the missing details. As shown in Table 3, DeepSeekMath-Base 7B demonstrates strong
performance in proof autoformalization.
Model
Size MMLU
BBH
HumanEval (Pass@1) MBPP (Pass@1)
Mistral
7B
62.4%
55.7%
28.0%
41.4%
DeepSeek-Coder-Base-v1.5† 7B
42.9%
42.9%
40.2%
52.6%
DeepSeek-Coder-Base-v1.5
7B
49.1%
55.2%
43.2%
60.4%
DeepSeekMath-Base
7B
54.9%
59.5%
40.9%
52.6%
Table 4 | Evaluation on natural language understanding, reasoning, and code benchmarks.
DeepSeek-Coder-Base-v1.5† is the checkpoint right before learning rate decay, which is used to
train DeepSeekMath-Base. On MMLU and BBH, we use few-shot chain-of-thought prompting.
On HumanEval and MBPP, we evaluate model performance under the zero-shot setting and a
few-shot setting, respectively.
Natural Language Understanding, Reasoning, and Code
We evaluate model performance of
natural language understanding on MMLU (Hendrycks et al., 2020), reasoning on BBH (Suzgun
et al., 2022), and coding capabilities on HumanEval (Chen et al., 2021) and MBPP (Austin et al.,
9


2021). As shown in Table 4, DeepSeekMath-Base 7B exhibits significant enhancements in per-
formance on MMLU and BBH over its precursor, DeepSeek-Coder-Base-v1.5 (Guo et al., 2024),
illustrating the positive impact of math training on language understanding and reasoning.
Additionally, by including code tokens for continual training, DeepSeekMath-Base 7B effectively
maintains the performance of DeepSeek-Coder-Base-v1.5 on the two coding benchmarks. Over-
all, DeepSeekMath-Base 7B significantly outperforms the general model Mistral 7B (Jiang et al.,
2023) on the three reasoning and coding benchmarks.
3. Supervised Fine-Tuning
3.1. SFT Data Curation
We construct a mathematical instruction-tuning dataset covering English and Chinese problems
from different mathematical fields and of varying complexity levels: problems are paired with
solutions in chain-of-thought (CoT) (Wei et al., 2022), program-of-thought (PoT) (Chen et al.,
2022; Gao et al., 2023), and tool-integrated reasoning format (Gou et al., 2023). The total number
of training examples is 776K.
• English mathematical datasets: We annotate GSM8K and MATH problems with tool-
integrated solutions, and adopt a subset of MathInstruct (Yue et al., 2023) along with the
training set of Lila-OOD (Mishra et al., 2022) where problems are solved with CoT or
PoT. Our English collection covers diverse fields of mathematics, e.g., algebra, probability,
number theory, calculus, and geometry.
• Chinese mathematical datasets: We collect Chinese K-12 mathematical problems spanning
76 sub-topics such as linear equations, with solutions annotated in both CoT and tool-
integrated reasoning format.
3.2. Training and Evaluating DeepSeekMath-Instruct 7B
In this section, we introduce DeepSeekMath-Instruct 7B which undergoes mathematical instruc-
tion tuning based on DeepSeekMath-Base. Training examples are randomly concatenated until
reaching a maximum context length of 4K tokens. We train the model for 500 steps with a batch
size of 256 and a constant learning rate of 5e-5.
We evaluate models’ mathematical performance both without and with tool use, on 4
quantitative reasoning benchmarks in English and Chinese. We benchmark our model against
the leading models of the time:
• Closed-source models include: (1) the GPT family among which GPT-4 (OpenAI, 2023)
and GPT-4 Code Interpreter 2 are the most capable ones, (2) Gemini Ultra and Pro (Anil
et al., 2023), (3) Inflection-2 (Inflection AI, 2023), (4) Grok-1 3, as well as models recently
released by Chinese companies including (5) Baichuan-3 4, (6) the latest GLM-4 5 from the
GLM family (Du et al., 2022). These models are for general purposes, most of which have
undergone a series of alignment procedures.
• Open-source models include: general models like (1) DeepSeek-LLM-Chat 67B (DeepSeek-
AI, 2024), (2) Qwen 72B (Bai et al., 2023), (3) SeaLLM-v2 7B (Nguyen et al., 2023), and (4)
2https://openai.com/blog/chatgpt-plugins#code-interpreter
3https://x.ai/model-card
4https://www.baichuan-ai.com
5https://open.bigmodel.cn/dev/api#glm-4
10


ChatGLM3 6B (ChatGLM3 Team, 2023), as well as models with enhancements in mathemat-
ics including (5) InternLM2-Math 20B 6 which builds on InternLM2 and underwent math
training followed by instruction tuning, (6) Math-Shepherd-Mistral 7B which applys PPO
training (Schulman et al., 2017) to Mistral 7B (Jiang et al., 2023) with a process-supervised
reward model, (7) the WizardMath series (Luo et al., 2023) which improves mathematical
reasoning in Mistral 7B and Llama-2 70B (Touvron et al., 2023) using evolve-instruct (i.e.,
a version of instruction tuning that uses AI-evolved instructions) and PPO training with
training problems primarily sourced from GSM8K and MATH, (8) MetaMath 70B (Yu et al.,
2023) which is Llama-2 70B fine-tuned on an augmented version of GSM8K and MATH,
(9) ToRA 34B Gou et al. (2023) which is CodeLlama 34B fine-tuned to do tool-integrated
mathematical reasoning, (10) MAmmoTH 70B (Yue et al., 2023) which is Llama-2 70B
instruction-tuned on MathInstruct.
As shown in Table 5, under the evaluation setting where tool use is disallowed, DeepSeekMath-
Instruct 7B demonstrates strong performance of step-by-step reasoning.
Notably, on the
competition-level MATH dataset, our model surpasses all open-source models and the ma-
jority of proprietary models (e.g., Inflection-2 and Gemini Pro) by at least 9% absolute. This
is true even for models that are substantially larger (e.g., Qwen 72B) or have been specifi-
cally enhanced through math-focused reinforcement learning (e.g., WizardMath-v1.1 7B). While
DeepSeekMath-Instruct rivals the Chinese proprietary models GLM-4 and Baichuan-3 on MATH,
it still underperforms GPT-4 and Gemini Ultra.
Under the evaluation setting where models are allowed to integrate natural language rea-
soning and program-based tool use for problem solving, DeepSeekMath-Instruct 7B approaches
an accuracy of 60% on MATH, surpassing all existing open-source models. On the other bench-
marks, our model is competitive with DeepSeek-LLM-Chat 67B, the prior state-of-the-art that is
10 times larger.
4. Reinforcement Learning
4.1. Group Relative Policy Optimization
Reinforcement learning (RL) has been proven to be effective in further improving the mathe-
matical reasoning ability of LLMs after the Supervised Fine-Tuning (SFT) stage (Luo et al., 2023;
Wang et al., 2023b). In this section, we introduce our efficient and effective RL algorithm, Group
Relative Policy Optimization (GRPO).
4.1.1. From PPO to GRPO
Proximal Policy Optimization (PPO) (Schulman et al., 2017) is an actor-critic RL algorithm that is
widely used in the RL fine-tuning stage of LLMs (Ouyang et al., 2022). In particular, it optimizes
LLMs by maximizing the following surrogate objective:
J
𝑃𝑃𝑂(𝜃) = E[𝑞∼𝑃(𝑄), 𝑜∼𝜋𝜃𝑜𝑙𝑑(𝑂|𝑞)] 1
|𝑜|
|𝑜|
∑︁
𝑡=1
min
 𝜋𝜃(𝑜𝑡|𝑞, 𝑜<𝑡)
𝜋𝜃𝑜𝑙𝑑(𝑜𝑡|𝑞, 𝑜<𝑡) 𝐴𝑡, clip
 𝜋𝜃(𝑜𝑡|𝑞, 𝑜<𝑡)
𝜋𝜃𝑜𝑙𝑑(𝑜𝑡|𝑞, 𝑜<𝑡) , 1 −𝜀, 1 + 𝜀

𝐴𝑡

,
(1)
where 𝜋𝜃and 𝜋𝜃𝑜𝑙𝑑are the current and old policy models, and 𝑞, 𝑜are questions and outputs
sampled from the question dataset and the old policy 𝜋𝜃𝑜𝑙𝑑, respectively. 𝜀is a clipping-related
hyper-parameter introduced in PPO for stabilizing training. 𝐴𝑡is the advantage, which is
computed by applying Generalized Advantage Estimation (GAE) (Schulman et al., 2015), based
6https://github.com/InternLM/InternLM-Math
11


Model
Size English Benchmarks Chinese Benchmarks
GSM8K
MATH
MGSM-zh CMATH
Chain-of-Thought Reasoning
Closed-Source Model
Gemini Ultra
-
94.4%
53.2%
-
-
GPT-4
-
92.0%
52.9%
-
86.0%
Inflection-2
-
81.4%
34.8%
-
-
GPT-3.5
-
80.8%
34.1%
-
73.8%
Gemini Pro
-
86.5%
32.6%
-
-
Grok-1
-
62.9%
23.9%
-
-
Baichuan-3
-
88.2%
49.2%
-
-
GLM-4
-
87.6%
47.9%
-
-
Open-Source Model
InternLM2-Math
20B
82.6%
37.7%
-
-
Qwen
72B
78.9%
35.2%
-
-
Math-Shepherd-Mistral
7B
84.1%
33.0%
-
-
WizardMath-v1.1
7B
83.2%
33.0%
-
-
DeepSeek-LLM-Chat
67B
84.1%
32.6%
74.0%
80.3%
MetaMath
70B
82.3%
26.6%
66.4%
70.9%
SeaLLM-v2
7B
78.2%
27.5%
64.8%
-
ChatGLM3
6B
72.3%
25.7%
-
-
WizardMath-v1.0
70B
81.6%
22.7%
64.8%
65.4%
DeepSeekMath-Instruct 7B
82.9%
46.8%
73.2%
84.6%
DeepSeekMath-RL
7B
88.2%
51.7%
79.6%
88.8%
Tool-Integrated Reasoning
Closed-Source Model
GPT-4 Code Interpreter
-
97.0%
69.7%
-
-
Open-Source Model
InternLM2-Math
20B
80.7%
54.3%
-
-
DeepSeek-LLM-Chat
67B
86.7%
51.1%
76.4%
85.4%
ToRA
34B
80.7%
50.8%
41.2%
53.4%
MAmmoTH
70B
76.9%
41.8%
-
-
DeepSeekMath-Instruct 7B
83.7%
57.4%
72.0%
84.3%
DeepSeekMath-RL
7B
86.7%
58.8%
78.4%
87.6%
Table 5 | Performance of Open- and Closed-Source models with both Chain-of-Thought and
Tool-Integrated Reasoning on English and Chinese Benchmarks. Scores in gray denote majority
votes with 32 candidates; The others are Top1 scores. DeepSeekMath-RL 7B beats all open-
source models from 7B to 70B, as well as the majority of closed-source models. Although
DeepSeekMath-RL 7B is only further trained on chain-of-thought-format instruction tuning data
of GSM8K and MATH, it improves over DeepSeekMath-Instruct 7B on all benchmarks.
12


𝑞𝑞
𝑜𝑜!
𝑜𝑜"
𝑜𝑜#
𝑟𝑟
!
𝑟𝑟"
𝑟𝑟#
𝐴𝐴!
𝐴𝐴"
𝐴𝐴#
𝑞𝑞
𝑜𝑜
GAE
𝐴𝐴
𝑟𝑟
𝑣𝑣
Reward 
Model
Policy 
Model
Value 
Model
…
…
…
Policy 
Model
Reference 
Model
Reward 
Model
PPO
GRPO
Trained
Models
Frozen
Models
Reference 
Model
⊕
𝐾𝐾𝐾𝐾
𝐾𝐾𝐾𝐾
Group 
Computation
Figure 4 | Demonstration of PPO and our GRPO. GRPO foregoes the value model, instead
estimating the baseline from group scores, significantly reducing training resources.
on the rewards {𝑟≥𝑡} and a learned value function 𝑉𝜓. Thus, in PPO, a value function needs to
be trained alongside the policy model and to mitigate over-optimization of the reward model,
the standard approach is to add a per-token KL penalty from a reference model in the reward at
each token (Ouyang et al., 2022), i.e.,
𝑟𝑡= 𝑟𝜑(𝑞, 𝑜≤𝑡) −𝛽log 𝜋𝜃(𝑜𝑡|𝑞, 𝑜<𝑡)
𝜋𝑟𝑒𝑓(𝑜𝑡|𝑞, 𝑜<𝑡) ,
(2)
where 𝑟𝜑is the reward model, 𝜋𝑟𝑒𝑓is the reference model, which is usually the initial SFT model,
and 𝛽is the coefficient of the KL penalty.
As the value function employed in PPO is typically another model of comparable size as
the policy model, it brings a substantial memory and computational burden. Additionally,
during RL training, the value function is treated as a baseline in the calculation of the advantage
for variance reduction. While in the LLM context, usually only the last token is assigned a
reward score by the reward model, which may complicate the training of a value function that is
accurate at each token. To address this, as shown in Figure 4, we propose Group Relative Policy
Optimization (GRPO), which obviates the need for additional value function approximation as
in PPO, and instead uses the average reward of multiple sampled outputs, produced in response
to the same question, as the baseline. More specifically, for each question 𝑞, GRPO samples a
group of outputs {𝑜1, 𝑜2, · · · , 𝑜𝐺} from the old policy 𝜋𝜃𝑜𝑙𝑑and then optimizes the policy model
by maximizing the following objective:
J
𝐺𝑅𝑃𝑂(𝜃) = E[𝑞∼𝑃(𝑄), {𝑜𝑖}𝐺
𝑖=1 ∼𝜋𝜃𝑜𝑙𝑑(𝑂|𝑞)]
1
𝐺
𝐺
∑︁
𝑖=1
1
|𝑜𝑖|
|𝑜𝑖|
∑︁
𝑡=1

min
 𝜋𝜃(𝑜𝑖,𝑡|𝑞, 𝑜𝑖,<𝑡)
𝜋𝜃𝑜𝑙𝑑(𝑜𝑖,𝑡|𝑞, 𝑜𝑖,<𝑡)
ˆ
𝐴𝑖,𝑡, clip
 𝜋𝜃(𝑜𝑖,𝑡|𝑞, 𝑜𝑖,<𝑡)
𝜋𝜃𝑜𝑙𝑑(𝑜𝑖,𝑡|𝑞, 𝑜𝑖,<𝑡) , 1 −𝜀, 1 + 𝜀

ˆ
𝐴𝑖,𝑡

−𝛽D𝐾𝐿

𝜋𝜃||𝜋𝑟𝑒𝑓

,
(3)
where 𝜀and 𝛽are hyper-parameters, and ˆ
𝐴𝑖,𝑡is the advantage calculated based on relative
rewards of the outputs inside each group only, which will be detailed in the following subsec-
tions. The group relative way that GRPO leverages to calculate the advantages, aligns well with
the comparative nature of rewards models, as reward models are typically trained on datasets
of comparisons between outputs on the same question. Also note that, instead of adding KL
penalty in the reward, GRPO regularizes by directly adding the KL divergence between the
trained policy and the reference policy to the loss, avoiding complicating the calculation of ˆ
𝐴𝑖,𝑡.
13


Algorithm 1 Iterative Group Relative Policy Optimization
Input initial policy model 𝜋𝜃init; reward models 𝑟𝜑; task prompts D; hyperparameters 𝜀, 𝛽, 𝜇
1: policy model 𝜋𝜃←𝜋𝜃init
2: for iteration = 1, ..., I do
3:
reference model 𝜋𝑟𝑒𝑓←𝜋𝜃
4:
for step = 1, ..., M do
5:
Sample a batch D𝑏from D
6:
Update the old policy model 𝜋𝜃𝑜𝑙𝑑←𝜋𝜃
7:
Sample 𝐺outputs {𝑜𝑖}𝐺
𝑖=1 ∼𝜋𝜃𝑜𝑙𝑑(· | 𝑞) for each question 𝑞∈D𝑏
8:
Compute rewards {𝑟𝑖}𝐺
𝑖=1 for each sampled output 𝑜𝑖by running 𝑟𝜑
9:
Compute ˆ
𝐴𝑖,𝑡for the 𝑡-th token of 𝑜𝑖through group relative advantage estimation.
10:
for GRPO iteration = 1, ..., 𝜇do
11:
Update the policy model 𝜋𝜃by maximizing the GRPO objective (Equation 21)
12:
Update 𝑟𝜑through continuous training using a replay mechanism.
Output 𝜋𝜃
And different from the KL penalty term used in (2), we estimate the KL divergence with the
following unbiased estimator (Schulman, 2020):
D𝐾𝐿

𝜋𝜃||𝜋𝑟𝑒𝑓

=
𝜋𝑟𝑒𝑓(𝑜𝑖,𝑡|𝑞, 𝑜𝑖,<𝑡)
𝜋𝜃(𝑜𝑖,𝑡|𝑞, 𝑜𝑖,<𝑡) −log
𝜋𝑟𝑒𝑓(𝑜𝑖,𝑡|𝑞, 𝑜𝑖,<𝑡)
𝜋𝜃(𝑜𝑖,𝑡|𝑞, 𝑜𝑖,<𝑡) −1,
(4)
which is guaranteed to be positive.
4.1.2. Outcome Supervision RL with GRPO
Formally, for each question 𝑞, a group of outputs {𝑜1, 𝑜2, · · · , 𝑜𝐺} are sampled from the old
policy model 𝜋𝜃𝑜𝑙𝑑. A reward model is then used to score the outputs, yielding 𝐺rewards
r = {𝑟1, 𝑟2, · · · , 𝑟𝐺} correspondingly. Subsequently, these rewards are normalized by subtracting
the group average and dividing by the group standard deviation. Outcome supervision provides
the normalized reward at the end of each output 𝑜𝑖and sets the advantages ˆ
𝐴𝑖,𝑡of all tokens in
the output as the normalized reward, i.e., ˆ
𝐴𝑖,𝑡= e
𝑟𝑖= 𝑟𝑖−mean(r)
std(r)
, and then optimizes the policy by
maximizing the objective defined in equation (3).
4.1.3. Process Supervision RL with GRPO
Outcome supervision only provides a reward at the end of each output, which may not be
sufficient and efficient to supervise the policy in complex mathematical tasks. Following Wang
et al. (2023b), we also explore process supervision, which provides a reward at the end of
each reasoning step. Formally, given the question 𝑞and 𝐺sampled outputs {𝑜1, 𝑜2, · · · , 𝑜𝐺}, a
process reward model is used to score each step of the outputs, yielding corresponding rewards:
R = {{𝑟𝑖𝑛𝑑𝑒𝑥(1)
1
, · · · , 𝑟𝑖𝑛𝑑𝑒𝑥(𝐾1)
1
}, · · · , {𝑟𝑖𝑛𝑑𝑒𝑥(1)
𝐺
, · · · , 𝑟𝑖𝑛𝑑𝑒𝑥(𝐾𝐺)
𝐺
}}, where 𝑖𝑛𝑑𝑒𝑥( 𝑗) is the end token index
of the 𝑗-th step, and 𝐾𝑖is the total number of steps in the 𝑖-th output. We also normalize these
rewards with the average and the standard deviation, i.e.,e
𝑟𝑖𝑛𝑑𝑒𝑥( 𝑗)
𝑖
=
𝑟𝑖𝑛𝑑𝑒𝑥( 𝑗)
𝑖
−mean(R)
std(R)
. Subsequently,
the process supervision calculates the advantage of each token as the sum of the normalized
rewards from the following steps, i.e., ˆ
𝐴𝑖,𝑡= Í
𝑖𝑛𝑑𝑒𝑥( 𝑗)≥𝑡e
𝑟𝑖𝑛𝑑𝑒𝑥( 𝑗)
𝑖
, and then optimizes the policy by
maximizing the objective defined in equation (3).
14


4.1.4. Iterative RL with GRPO
As the reinforcement learning training process progresses, the old reward model may not be
sufficient to supervise the current policy model. Therefore, we also explore the iterative RL
with GRPO. As shown in Algorithm 1, in iterative GRPO, we generate new training sets for the
reward model based on the sampling results from the policy model and continually train the
old reward model using a replay mechanism that incorporates 10% of historical data. Then, we
set the reference model as the policy model, and continually train the policy model with the
new reward model.
4.2. Training and Evaluating DeepSeekMath-RL
We conduct RL based on DeepSeekMath-Instruct 7B. The training data of RL are chain-of-
thought-format questions related to GSM8K and MATH from the SFT data, which consists
of around 144K questions. We exclude other SFT questions to investigate the impact of RL
on benchmarks that lack data throughout the RL phase. We construct the training set of
reward models following (Wang et al., 2023b). We train our initial reward model based on the
DeepSeekMath-Base 7B with a learning rate of 2e-5. For GRPO, we set the learning rate of the
policy model as 1e-6. The KL coefficient is 0.04. For each question, we sample 64 outputs. The
max length is set to 1024, and the training batch size is 1024. The policy model only has a single
update following each exploration stage. We evaluate DeepSeekMath-RL 7B on benchmarks
following DeepSeekMath-Instruct 7B. For DeepSeekMath-RL 7B, GSM8K and MATH with
chain-of-thought reasoning can be regarded as in-domain tasks and all the other benchmarks
can be regarded as out-of-domain tasks.
Table 5 demonstrates the performance of open- and closed-source models with both chain-
of-thought and tool-integrated reasoning on English and Chinese benchmarks. We find that:
1) DeepSeekMath-RL 7B attains accuracies of 88.2% and 51.7% on GSM8K and MATH, respec-
tively, utilizing chain-of-thought reasoning. This performance surpasses that of all open-source
models in the 7B to 70B range, as well as the majority of closed-source models. 2) Crucially,
DeepSeekMath-RL 7B is only trained on chain-of-thought-format instruction tuning data of
GSM8K and MATH, starting from DeepSeekMath-Instruct 7B. Despite the constrained scope
of its training data, it outperforms DeepSeekMath-Instruct 7B across all evaluation metrics,
showcasing the effectiveness of reinforcement learning.
5. Discussion
In this section, we will share our findings in pre-training and RL experiments.
5.1. Lessons Learnt in Pre-Training
We first share our experience in pre-training. Unless otherwise specified, we will adhere to
the training settings outlined in Section 2.2.1. It is worth noting that, when referring to the
DeepSeekMath Corpus in this section, we use an 89B-token dataset from the second iteration of
the data collection process.
5.1.1. Code Training Benefits Mathematical Reasoning
A popular yet unverified hypothesis suggests that code training improves reasoning. We attempt
to offer a partial response to this, particularly within the mathematical domain: code training
15


Training Setting
Training Tokens
w/o Tool Use
w/ Tool Use
General Code Math GSM8K MATH CMATH GSM8K+Python MATH+Python
No Continual Training
–
–
–
2.9%
3.0%
12.3%
2.7%
2.3%
Two-Stage Training
Stage 1: General Training
400B
–
–
2.9%
3.2%
14.8%
3.3%
2.3%
Stage 2: Math Training
–
–
150B
19.1%
14.4%
37.2%
14.3%
6.7%
Stage 1: Code Training
–
400B –
5.9%
3.6%
19.9%
12.4%
10.0%
Stage 2: Math Training
–
–
150B
21.9%
15.3%
39.7%
17.4%
9.4%
One-Stage Training
Math Training
–
–
150B
20.5%
13.1%
37.6%
11.4%
6.5%
Code & Math Mixed Training –
400B 150B
17.6%
12.1%
36.3%
19.7%
13.5%
Table 6 | Investigation of how code affects mathematical reasoning under different training
settings. We experiment with DeepSeek-LLM 1.3B, and evaluate its mathematical reasoning
performance without and with tool use via few-shot chain-of-thought prompting and few-shot
program-of-thought prompting, respectively.
improves models’ ability to do mathematical reasoning both with and without tool use.
To study how code training affects mathematical reasoning, we experimented with the
following two-stage training and one-stage training settings:
Two-Stage Training
• Code Training for 400B Tokens →Math Training for 150B Tokens: We train DeepSeek-
LLM 1.3B for 400B code tokens followed by 150B math tokens;
• General Training for 400B Tokens →Math Training for 150B Tokens: As a control
experiment, we also experiment with general tokens (sampled from a large-scale general
corpus created by DeepSeek-AI) instead of code tokens in the first stage of training, in an
attempt to investigate the advantages of code tokens over general tokens in improving
mathematical reasoning.
One-Stage Training
• Math Training for 150B Tokens: We train DeepSeek-LLM 1.3B for 150B math tokens;
• Training on a mixture of 400B Code Tokens and 150B Math Tokens: Math training fol-
lowing code training degrades coding performance. We investigate whether code tokens,
when mixed with math tokens for one-stage training, would still improve mathematical
reasoning and also alleviate the problem of catastrophic forgetting.
Results
Table 6 and Table 7 demonstrate the downstream performance under different training
settings.
Code training benefits program-aided mathematical reasoning, both under the two-stage
training and one-stage training settings. As shown in Table 6, under the two-stage training
setting, code training alone already significantly enhances the ability to solve GSM8K and
MATH problems using Python. Math training in the second stage yields further improvements.
Interestingly, under the one-stage training setting, mixing code tokens and math tokens effec-
tively mitigates the issue of catastrophic forgetting that arises from two-stage training, and also
synergizes coding (Table 7) and program-aided mathematical reasoning (Table 6).
16


Training Setting
Training Tokens
MMLU
BBH
HumanEval (Pass@1) MBPP (Pass@1)
General Code Math
No Continual Training
–
–
–
24.5%
28.1%
12.2%
13.0%
Two-Stage Training
Stage 1: General Training
400B
–
–
25.9%
27.7%
15.2%
13.6%
Stage 2: Math Training
–
–
150B
33.1%
32.7%
12.8%
13.2%
Stage 1: Code Training
–
400B –
25.0%
31.5%
25.0%
40.0%
Stage 2: Math Training
–
–
150B
36.2%
35.3%
12.2%
17.0%
One-Stage Training
Math Training
–
–
150B
32.3%
32.5%
11.6%
13.2%
Code & Math Mixed Training –
400B 150B
33.5%
35.6%
29.3%
39.4%
Table 7 | Investigation of how different settings of code and math training affect model perfor-
mance of language understanding, reasoning, and coding. We experiment with DeepSeek-LLM
1.3B. We evaluate the models on MMLU and BBH using few-shot chain-of-thought prompting.
On HumanEval and MBPP, we conduct zero-shot and few-shot evaluations, respectively.
Model
Size ArXiv Corpus
English Benchmarks
Chinese Benchmarks
GSM8K MATH OCW
SAT
MMLU
STEM
CMATH
Gaokao
MathCloze
Gaokao
MathQA
DeepSeek-LLM
1.3B
No Math Training
2.9%
3.0%
2.9% 15.6%
19.5%
12.3%
0.8%
17.9%
MathPile
2.7%
3.3%
2.2% 12.5%
15.7%
1.2%
0.0%
2.8%
ArXiv-RedPajama
3.3%
3.4%
4.0%
9.4%
9.0%
7.4%
0.8%
2.3%
DeepSeek-Coder-Base-v1.5 7B
No Math Training
29.0%
12.5%
6.6% 40.6%
38.1%
45.9%
5.9%
21.1%
MathPile
23.6%
11.5%
7.0% 46.9%
35.8%
37.9%
4.2%
25.6%
ArXiv-RedPajama
28.1%
11.1%
7.7% 50.0%
35.2%
42.6%
7.6%
24.8%
Table 8 | Effect of math training on different arXiv datasets. Model performance is evaluated
with few-shot chain-of-thought prompting.
ArXiv Corpus
miniF2F-valid miniF2F-test
No Math Training
20.1%
21.7%
MathPile
16.8%
16.4%
ArXiv-RedPajama
14.8%
11.9%
Table 9 | Effect of math training on different arXiv corpora, the base model being DeepSeek-
Coder-Base-v1.5 7B. We evaluate informal-to-formal proving in Isabelle.
Code training also improves mathematical reasoning without tool use. Under the two-stage
training setting, the initial stage of code training already results in moderate enhancements.
It also boosts the efficiency of the subsequent math training, eventually leading to the best
performance. However, combining code tokens and math tokens for one-stage training com-
promises mathematical reasoning without tool use. One conjecture is that DeepSeek-LLM 1.3B,
due to its limited scale, lacks the capacity to fully assimilate both code and mathematical data
simultaneously.
5.1.2. ArXiv Papers Seem Ineffective in Improving Mathematical Reasoning
ArXiv papers are commonly included as a component of math pre-training data (Azerbayev
et al., 2023; Lewkowycz et al., 2022a; Polu and Sutskever, 2020; Wang et al., 2023c). However,
17


detailed analysis regarding their impact on mathematical reasoning has not been extensively
conducted. Perhaps counter-intuitively, according to our experiments, arXiv papers seem
ineffective in improving mathematical reasoning. We experiment with models of different sizes,
including DeepSeek-LLM 1.3B and DeepSeek-Coder-Base-v1.5 7B (Guo et al., 2024), using arXiv
corpora that underwent varied processing pipelines:
• MathPile (Wang et al., 2023c): an 8.9B-token corpus developed with cleaning and filtering
heuristic rules, over 85% of which are scientific arXiv papers;
• ArXiv-RedPajama (Computer, 2023): the entirety of arXiv LaTeX files with preambles,
comments, macros, and bibliographies removed, totaling 28.0B tokens.
In our experiments, we separately train DeepSeek-LLM 1.3B for 150B tokens and DeepSeek-
Coder-Base-v1.5 7B for 40B tokens on each arXiv corpus. It seems that arXiv papers are ineffective
in improving mathematical reasoning. When trained on a arXiv-only corpus, both models dis-
play no notable improvements or even deterioration across various mathematical benchmarks of
different complexities employed in this study. These benchmarks include quantitative reasoning
datasets like GSM8K and MATH (Table 8), multiple-choice challenges like MMLU-STEM (Table
8), and formal mathematics like miniF2F (Table 9).
However, this conclusion has its limitations and should be taken with a grain of salt. We
have not yet studied:
• The impact of arXiv tokens on specific math-related tasks not included in this research,
such as informalization of theorems which is to convert formal statements or proofs to
their informal versions;
• The effect of arXiv tokens when combined with other types of data;
• Whether the benefits of arXiv papers would manifest themselves at a larger model scale.
Thus, further exploration is required, which we leave for future studies.
5.2. Insights of Reinforcement Learning
5.2.1. Towards to a Unified Paradigm
In this section, we provide a unified paradigm to analyze different training methods, such as
SFT, RFT, DPO, PPO, GRPO, and further conduct experiments to explore the factors of the
unified paradigm. Generally, the gradient with respect to the parameter 𝜃of a training method
can be written as:
∇𝜃JA(𝜃) = E[(𝑞, 𝑜) ∼D
|       {z       }
𝐷𝑎𝑡𝑎𝑆𝑜𝑢𝑟𝑐𝑒
]
©
­
­
­
«
1
|𝑜|
|𝑜|
∑︁
𝑡=1
𝐺𝐶A(𝑞, 𝑜, 𝑡, 𝜋𝑟𝑓)
|               {z               }
𝐺𝑟𝑎𝑑𝑖𝑒𝑛𝑡𝐶𝑜𝑒𝑓𝑓𝑖𝑐𝑖𝑒𝑛𝑡
∇𝜃log 𝜋𝜃(𝑜𝑡|𝑞, 𝑜<𝑡)
ª
®
®
®
¬
.
(5)
There exist three key components: 1) Data Source D, which determines the training data; 2)
Reward Function 𝜋𝑟𝑓, which is the source of the training reward signal; 3) Algorithm A: which
processes the training data and the reward signal to the gradient coefficient 𝐺𝐶that determines
the magnitude of the penalty or reinforcement for the data. We analyze several representative
methods based on such a unified paradigm:
• Supervised Fine-tuning (SFT): SFT fine-tunes pretrained model on human selected SFT
data.
18


Methods
Data Source
Reward Function
Gradient Coefficient
SFT
𝑞, 𝑜∼𝑃𝑠𝑓𝑡(𝑄, 𝑂)
-
1
RFT
𝑞∼𝑃𝑠𝑓𝑡(𝑄), 𝑜∼𝜋𝑠𝑓𝑡(𝑂|𝑞)
Rule
Equation 10
DPO
𝑞∼𝑃𝑠𝑓𝑡(𝑄), 𝑜+, 𝑜−∼𝜋𝑠𝑓𝑡(𝑂|𝑞)
Rule
Equation 14
Online RFT
𝑞∼𝑃𝑠𝑓𝑡(𝑄), 𝑜∼𝜋𝜃(𝑂|𝑞)
Rule
Equation 10
PPO
𝑞∼𝑃𝑠𝑓𝑡(𝑄), 𝑜∼𝜋𝜃(𝑂|𝑞)
Model
Equation 18
GRPO
𝑞∼𝑃𝑠𝑓𝑡(𝑄), {𝑜𝑖}𝐺
𝑖=1 ∼𝜋𝜃(𝑂|𝑞)
Model
Equation 21
Table 10 | The data source and gradient coefficient of different methods. 𝑃𝑠𝑓𝑡denotes the data
distribution of supervised fine-tuning datasets. 𝜋𝜃𝑠𝑓𝑡and 𝜋𝜃denote the supervised fine-tuned
model and the real-time policy model during the online training process, respectively.
0
2000
4000
6000
8000
Steps
56
58
60
62
64
66
Acc (%)
GSM8K
0
2000
4000
6000
8000
Steps
27
28
29
30
Acc (%)
MATH
RFT
Online RFT
GRPO+OS
GRPO+PS
Figure 5 | Performance of the DeepSeekMath-Instruct 1.3B model, which was further trained
using various methods, on two benchmarks.
• Rejection Sampling Fine-tuning (RFT): RFT further fine-tunes the SFT model on the
filtered outputs sampled from the SFT model based on SFT questions. RFT filters the
outputs based on the correctness of their answers.
• Direct Preference Optimization (DPO): DPO further refines the SFT model by fine-tuning
it on augmented outputs sampled from the SFT model, using pair-wise DPO loss.
• Online Rejection Sampling Fine-tuning (Online RFT): Different from RFT, Online RFT
initiates the policy model using the SFT model and refines it by fine-tuning with the
augmented outputs sampled from the real-time policy model.
• PPO/GRPO: PPO/GRPO initializes the policy model using the SFT model and reinforces
it with the outputs sampled from the real-time policy model.
We summarize the components of these methods in Table 10. Please refer to Appendix A.1 for a
more detailed derivation process.
Observation about Data Source
We divide the data source into two categories, online sam-
pling, and offline sampling. Online sampling denotes that the training data is from the explo-
ration results of the real-time training policy model, while offline sampling denotes that the
19


0
1300
2300
3300
4300
5300
Steps
83
84
85
86
87
88
89
Acc (%)
GSM8K
0
1300
2300
3300
4300
5300
Steps
47
48
49
50
51
52
Acc (%)
MATH
Iteration-0
Iteration-1
Iteration-2
Figure 6 | Performance of iterative reinforcement learning with DeepSeekMath-Instruct 7B on
two benchmarks.
training data is from the sampling results of the initial SFT model. RFT and DPO follow the
offline style, while Online RFT and GRPO follow the online style.
As shown in Figure 5, we find that the Online RFT significantly outperforms RFT on two
benchmarks. Specifically, Online RFT is comparable to RFT in the early stage of training but
gains an absolute advantage in the later stage, demonstrating the superiority of online training.
This is intuitive, as in the initial stage, the actor and the SFT model exhibit close resemblance,
with the sampled data revealing only minor differences. In the later stage, however, the data
sampled from the actor will exhibit more significant differences, and real-time data sampling
will offer greater advantages.
Observation about Gradient Coefficient
The algorithm processes the reward signal to the
gradient coefficient to update the model parameter. We divide the reward function as ‘Rule’
and ‘Model’ in our experiments. Rule refers to judging the quality of a response based on
the correctness of the answer, and Model denotes that we train a reward model to score each
response. The training data of the reward model is based on the rule judgment. Equations 10
and 21 highlight a key difference between GRPO and Online RFT: GRPO uniquely adjusts its
gradient coefficient based on the reward value provided by the reward model. This allows for
differential reinforcement and penalization of responses according to their varying magnitudes.
In contrast, Online RFT lacks this feature; it does not penalize incorrect responses and uniformly
reinforces all responses with correct answers at the same level of intensity.
As demonstrated in Figure 5, GRPO surpasses online RFT, thereby highlighting the efficiency
of altering positive and negative gradient coefficients. In addition, GRPO+PS shows superior
performance compared to GRPO+OS, indicating the benefits of using fine-grained, step-aware
gradient coefficients. Furthermore, we explore the iterative RL, in our experiments, we conduct
two rounds of iteration. As shown in Figure 6, we notice that the iterative RL significantly
improves the performance, especially at the first iteration.
20


1
4
8
16
32
64
K: The number of candidates
82
84
86
88
90
92
94
96
98
Acc (%)
GSM8K
1
4
8
16
32
64
K: The number of candidates
45
50
55
60
65
70
75
80
85
Acc (%)
MATH
Maj@K-Instruct
Maj@K-RL
Pass@K-Instruct
Pass@K-RL
Figure 7 | The Maj@K and Pass@K of SFT and RL DeepSeekMath 7B on GSM8K and MATH
(temperature 0.7). It was noted that RL enhances Maj@K but not Pass@K.
5.2.2. Why RL Works?
In this paper, we conduct reinforcement learning based on a subset of instruction tuning
data, and it achieves significant performance enhancement upon the instruction tuning model.
To further explain why reinforcement learning works. We evaluate the Pass@K and Maj@K
accuracy of the Instruct and RL models on two benchmarks. As shown in Figure 7, RL enhances
Maj@K’s performance but not Pass@K. These findings indicate that RL enhances the model’s
overall performance by rendering the output distribution more robust, in other words, it seems
that the improvement is attributed to boosting the correct response from TopK rather than
the enhancement of fundamental capabilities. Similarly, (Wang et al., 2023a) identified a
misalignment problem in reasoning tasks within the SFT model, showing that the reasoning
performance of SFT models can be improved through a series of preference alignment strategies
(Song et al., 2023; Wang et al., 2023a; Yuan et al., 2023b).
5.2.3. How to Achieve More Effective RL?
We demonstrate RL works pretty well in mathematical reasoning tasks. We also provide a unified
paradigm to understand different representative training methods. Within this paradigm, all
methods are conceptualized as either direct or simplified RL techniques. As summarized in
Equation 5, there exist three key components: Data Source, Algorithm, and Reward Function.
We provide some potential future directions about the three components.
Data Source
Data source is the raw material of all training methods. In the context of RL, we
specifically refer to the data source as the unlabeled questions with the outputs sampled from
the policy model. In this paper, we only use the questions from the instruction tuning stage and
a naive nucleus sampling to sample outputs. We think this is a potential reason that our RL
pipeline only improves the Maj@K performance. In the future, we will explore our RL pipeline
on out-of-distribution question prompts, in conjunction with advanced sampling (decoding)
strategies, like those based on tree-search methods (Yao et al., 2023). Also, the efficient inference
techniques (Kwon et al., 2023; Leviathan et al., 2023; Xia et al., 2023, 2024), which determines
21


the exploration efficiency of policy models, also play an exceedingly important role.
Algorithms
Algorithms process the data and reward signal to the gradient coefficient to update
the model parameter. Based on Equation 5, to some extent, all methods now fully TRUST the
signal of the reward function to increase or decrease the conditional probability of a certain
token. However, it is impossible to ensure the reward signal is always reliable, especially in
extremely complex tasks. For example, even the PRM800K datasets (Lightman et al., 2023),
which have been carefully annotated by well-trained annotators, still contain approximately 20%
of incorrectly annotations7. To this end, we will explore the reinforcement learning algorithm
that is robust against noisy reward signals. We believe such WEAK-TO-STRONG (Burns et al.,
2023) alignment methods will bring a fundamental change to the learning algorithms.
Reward Function
Reward function is the source of the training signal. In RL, the reward
function is usually the neural reward model. We think there exist three important directions for
reward models: 1) How to enhance the generalization ability of the reward model. The reward
model must be effectively generalized to handle out-of-distribution questions and advanced
decoding outputs; otherwise, reinforcement learning may merely stabilize the distribution of
LLMs rather than improve their fundamental capabilities; 2) How to reflect the uncertainty
of reward model. The uncertainty could potentially act as a linking bridge between the weak
reward model and the weak-to-strong learning algorithms; 3) How to efficiently build high-
quality process reward models that can provide fine-grained training signals for the reasoning
process (Lightman et al., 2023; Wang et al., 2023b).
6. Conclusion, Limitation, and Future Work
We present DeepSeekMath, which outperforms all open-source models on the competition-
level MATH benchmark and approaches the performance of closed models. DeepSeekMath is
initialized with DeepSeek-Coder-v1.5 7B and undergoes continual training for 500B tokens, with
a significant component of the training data being 120B math tokens sourced from Common
Crawl. Our extensive ablation study shows web pages offer significant potential for high-quality
mathematical data, while arXiv may not as beneficial as we expected. We introduce Group
Relative Policy Optimization (GRPO), a variant of Proximal Policy Optimization (PPO), which
can notably improve mathematical reasoning capabilities with less memory consumption. The
experiment results show that GRPO is effective even if DeepSeekMath-Instruct 7B has reached
a high score on benchmarks. We also provide a unified paradigm to understand a series of
methods and summarize several potential directions for more effective reinforcement learning.
Although DeepSeekMath achieves impressive scores on quantitative reasoning benchmarks,
its capability on geometry and theorem-proof are relatively weaker than closed models. For
instance, in our dry run, the model cannot handle problems related to triangles and ellipses,
which may indicate data selection bias in pre-training and fine-tuning. In addition, restricted
by the model scale, DeepSeekMath is worse than GPT-4 on few-shot capability. GPT-4 could
improve its performance with few-shot inputs, while DeepSeekMath shows similar performance
in zero-shot and few-shot evaluation. In the future, we will further improve our engineered
data selection pipeline to construct more high-quality pre-trained corpus. In addition, we will
explore the potential directions (Section 5.2.3) for more effective reinforcement learning of LLMs.
7https://github.com/openai/prm800k/issues/12#issuecomment-1728491852
22


References
R. Anil, S. Borgeaud, Y. Wu, J. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth,
K. Millican, D. Silver, S. Petrov, M. Johnson, I. Antonoglou, J. Schrittwieser, A. Glaese, J. Chen,
E. Pitler, T. P. Lillicrap, A. Lazaridou, O. Firat, J. Molloy, M. Isard, P. R. Barham, T. Hennigan,
B. Lee, F. Viola, M. Reynolds, Y. Xu, R. Doherty, E. Collins, C. Meyer, E. Rutherford, E. Moreira,
K. Ayoub, M. Goel, G. Tucker, E. Piqueras, M. Krikun, I. Barr, N. Savinov, I. Danihelka,
B. Roelofs, A. White, A. Andreassen, T. von Glehn, L. Yagati, M. Kazemi, L. Gonzalez,
M. Khalman, J. Sygnowski, and et al. Gemini: A family of highly capable multimodal
models. CoRR, abs/2312.11805, 2023. doi: 10.48550/ARXIV.2312.11805. URL https:
//doi.org/10.48550/arXiv.2312.11805.
J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry,
Q. Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732,
2021.
Z. Azerbayev, H. Schoelkopf, K. Paster, M. D. Santos, S. McAleer, A. Q. Jiang, J. Deng, S. Bider-
man, and S. Welleck. Llemma: An open language model for mathematics. arXiv preprint
arXiv:2310.10631, 2023.
J. Bai, S. Bai, Y. Chu, Z. Cui, K. Dang, X. Deng, Y. Fan, W. Ge, Y. Han, F. Huang, et al. Qwen
technical report. arXiv preprint arXiv:2309.16609, 2023.
C. Burns, P. Izmailov, J. H. Kirchner, B. Baker, L. Gao, L. Aschenbrenner, Y. Chen, A. Ecoffet,
M. Joglekar, J. Leike, et al. Weak-to-strong generalization: Eliciting strong capabilities with
weak supervision. arXiv preprint arXiv:2312.09390, 2023.
ChatGLM3 Team. Chatglm3 series: Open bilingual chat llms, 2023. URL https://github.c
om/THUDM/ChatGLM3.
M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda,
N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin,
B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet,
F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herbert-Voss, W. H. Guss,
A. Nichol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse,
A. N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford, M. Knight, M. Brundage,
M. Murati, K. Mayer, P. Welinder, B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, and
W. Zaremba. Evaluating large language models trained on code. CoRR, abs/2107.03374, 2021.
URL https://arxiv.org/abs/2107.03374.
W. Chen, X. Ma, X. Wang, and W. W. Cohen. Program of thoughts prompting: Disentangling
computation from reasoning for numerical reasoning tasks. CoRR, abs/2211.12588, 2022. doi:
10.48550/ARXIV.2211.12588. URL https://doi.org/10.48550/arXiv.2211.12588.
K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek,
J. Hilton, R. Nakano, et al. Training verifiers to solve math word problems. arXiv preprint
arXiv:2110.14168, 2021.
T. Computer. Redpajama: an open dataset for training large language models, Oct. 2023. URL
https://github.com/togethercomputer/RedPajama-Data.
DeepSeek-AI. Deepseek LLM: scaling open-source language models with longtermism. CoRR,
abs/2401.02954, 2024. doi: 10.48550/ARXIV.2401.02954. URL https://doi.org/10.485
50/arXiv.2401.02954.
23


Z. Du, Y. Qian, X. Liu, M. Ding, J. Qiu, Z. Yang, and J. Tang. Glm: General language model
pretraining with autoregressive blank infilling. In Proceedings of the 60th Annual Meeting
of the Association for Computational Linguistics (Volume 1: Long Papers), pages 320–335,
2022.
L. Gao, A. Madaan, S. Zhou, U. Alon, P. Liu, Y. Yang, J. Callan, and G. Neubig. PAL: program-
aided language models. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and
J. Scarlett, editors, International Conference on Machine Learning, ICML 2023, 23-29 July
2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research,
pages 10764–10799. PMLR, 2023. URL https://proceedings.mlr.press/v202/gao23f.
html.
Z. Gou, Z. Shao, Y. Gong, Y. Shen, Y. Yang, M. Huang, N. Duan, and W. Chen. Tora: A tool-
integrated reasoning agent for mathematical problem solving. CoRR, abs/2309.17452, 2023.
doi: 10.48550/ARXIV.2309.17452. URL https://doi.org/10.48550/arXiv.2309.1745
2.
D. Guo, Q. Zhu, D. Yang, Z. Xie, K. Dong, W. Zhang, G. Chen, X. Bi, Y. Wu, Y. K. Li, F. Luo,
Y. Xiong, and W. Liang. Deepseek-coder: When the large language model meets programming
– the rise of code intelligence, 2024.
D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Measuring
massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020.
D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt. Mea-
suring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874,
2021.
High-flyer. Hai-llm: 高效且轻量的大模型训练工具, 2023. URL https://www.high-flyer.c
n/en/blog/hai-llm.
Inflection AI. Inflection-2, 2023. URL https://inflection.ai/inflection-2.
A. Q. Jiang, S. Welleck, J. P. Zhou, W. Li, J. Liu, M. Jamnik, T. Lacroix, Y. Wu, and G. Lample. Draft,
sketch, and prove: Guiding formal theorem provers with informal proofs. arXiv preprint
arXiv:2210.12283, 2022.
A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. d. l. Casas, F. Bressand,
G. Lengyel, G. Lample, L. Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023.
A. Joulin, E. Grave, P. Bojanowski, M. Douze, H. Jégou, and T. Mikolov. Fasttext. zip: Compress-
ing text classification models. arXiv preprint arXiv:1612.03651, 2016.
W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, and I. Stoica.
Efficient memory management for large language model serving with pagedattention. In
Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Y. Leviathan, M. Kalman, and Y. Matias. Fast inference from transformers via speculative
decoding. In International Conference on Machine Learning, pages 19274–19286. PMLR,
2023.
A. Lewkowycz, A. Andreassen, D. Dohan, E. Dyer, H. Michalewski, V. Ramasesh, A. Slone,
C. Anil, I. Schlag, T. Gutman-Solo, et al. Solving quantitative reasoning problems with
language models. Advances in Neural Information Processing Systems, 35:3843–3857, 2022a.
24


A. Lewkowycz, A. Andreassen, D. Dohan, E. Dyer, H. Michalewski, V. V. Ramasesh, A. Slone,
C. Anil, I. Schlag, T. Gutman-Solo, Y. Wu, B. Neyshabur, G. Gur-Ari, and V. Misra. Solving
quantitative reasoning problems with language models. In S. Koyejo, S. Mohamed, A. Agarwal,
D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems
35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New
Orleans, LA, USA, November 28 - December 9, 2022, 2022b. URL http://papers.nips.
cc/paper_files/paper/2022/hash/18abbeef8cfe9203fdf9053c9c4fe191-Abstr
act-Conference.html.
H. Lightman, V. Kosaraju, Y. Burda, H. Edwards, B. Baker, T. Lee, J. Leike, J. Schulman,
I. Sutskever, and K. Cobbe. Let’s verify step by step. arXiv preprint arXiv:2305.20050, 2023.
I. Loshchilov and F. Hutter.
Decoupled weight decay regularization.
arXiv preprint
arXiv:1711.05101, 2017.
H. Luo, Q. Sun, C. Xu, P. Zhao, J. Lou, C. Tao, X. Geng, Q. Lin, S. Chen, and D. Zhang.
Wizardmath: Empowering mathematical reasoning for large language models via reinforced
evol-instruct. arXiv preprint arXiv:2308.09583, 2023.
S. Mishra, M. Finlayson, P. Lu, L. Tang, S. Welleck, C. Baral, T. Rajpurohit, O. Tafjord, A. Sab-
harwal, P. Clark, and A. Kalyan. LILA: A unified benchmark for mathematical reasoning.
In Y. Goldberg, Z. Kozareva, and Y. Zhang, editors, Proceedings of the 2022 Conference on
Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab
Emirates, December 7-11, 2022, pages 5807–5832. Association for Computational Linguistics,
2022. doi: 10.18653/V1/2022.EMNLP-MAIN.392. URL https://doi.org/10.18653/v1/
2022.emnlp-main.392.
X. Nguyen, W. Zhang, X. Li, M. M. Aljunied, Q. Tan, L. Cheng, G. Chen, Y. Deng, S. Yang,
C. Liu, H. Zhang, and L. Bing. Seallms - large language models for southeast asia. CoRR,
abs/2312.00738, 2023. doi: 10.48550/ARXIV.2312.00738. URL https://doi.org/10.485
50/arXiv.2312.00738.
OpenAI. GPT4 technical report. arXiv preprint arXiv:2303.08774, 2023.
L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal,
K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback.
Advances in Neural Information Processing Systems, 35:27730–27744, 2022.
K. Paster, M. D. Santos, Z. Azerbayev, and J. Ba. Openwebmath: An open dataset of high-quality
mathematical web text. CoRR, abs/2310.06786, 2023. doi: 10.48550/ARXIV.2310.06786. URL
https://doi.org/10.48550/arXiv.2310.06786.
L. C. Paulson. Three years of experience with sledgehammer, a practical link between auto-
matic and interactive theorem provers. In R. A. Schmidt, S. Schulz, and B. Konev, editors,
Proceedings of the 2nd Workshop on Practical Aspects of Automated Reasoning, PAAR-2010,
Edinburgh, Scotland, UK, July 14, 2010, volume 9 of EPiC Series in Computing, pages 1–10.
EasyChair, 2010. doi: 10.29007/TNFD. URL https://doi.org/10.29007/tnfd.
S. Polu and I. Sutskever. Generative language modeling for automated theorem proving. CoRR,
abs/2009.03393, 2020. URL https://arxiv.org/abs/2009.03393.
R. Rafailov, A. Sharma, E. Mitchell, S. Ermon, C. D. Manning, and C. Finn. Direct preference
optimization: Your language model is secretly a reward model. 2023.
25


J. Schulman. Approximating kl divergence, 2020. URL http://joschu.net/blog/kl-app
rox.html.
J. Schulman, P. Moritz, S. Levine, M. Jordan, and P. Abbeel. High-dimensional continuous
control using generalized advantage estimation. arXiv preprint arXiv:1506.02438, 2015.
J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization
algorithms. arXiv preprint arXiv:1707.06347, 2017.
F. Shi, M. Suzgun, M. Freitag, X. Wang, S. Srivats, S. Vosoughi, H. W. Chung, Y. Tay, S. Ruder,
D. Zhou, D. Das, and J. Wei. Language models are multilingual chain-of-thought reasoners.
In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali,
Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL https://openreview.net/pdf?id=
fR3wGCk-IXp.
F. Song, B. Yu, M. Li, H. Yu, F. Huang, Y. Li, and H. Wang. Preference ranking optimization for
human alignment. arXiv preprint arXiv:2306.17492, 2023.
M. Suzgun, N. Scales, N. Schärli, S. Gehrmann, Y. Tay, H. W. Chung, A. Chowdhery, Q. V. Le,
E. H. Chi, D. Zhou, et al. Challenging big-bench tasks and whether chain-of-thought can solve
them. arXiv preprint arXiv:2210.09261, 2022.
T. Tao. Embracing change and resetting expectations, 2023. URL https://unlocked.micro
soft.com/ai-anthology/terence-tao/.
H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra,
P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. Canton-Ferrer, M. Chen, G. Cucurull, D. Esiobu,
J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal, A. Hartshorn, S. Hosseini,
R. Hou, H. Inan, M. Kardas, V. Kerkez, M. Khabsa, I. Kloumann, A. Korenev, P. S. Koura,
M. Lachaux, T. Lavril, J. Lee, D. Liskovich, Y. Lu, Y. Mao, X. Martinet, T. Mihaylov, P. Mishra,
I. Molybog, Y. Nie, A. Poulton, J. Reizenstein, R. Rungta, K. Saladi, A. Schelten, R. Silva, E. M.
Smith, R. Subramanian, X. E. Tan, B. Tang, R. Taylor, A. Williams, J. X. Kuan, P. Xu, Z. Yan,
I. Zarov, Y. Zhang, A. Fan, M. Kambadur, S. Narang, A. Rodriguez, R. Stojnic, S. Edunov, and
T. Scialom. Llama 2: Open foundation and fine-tuned chat models. CoRR, abs/2307.09288,
2023. doi: 10.48550/arXiv.2307.09288. URL https://doi.org/10.48550/arXiv.2307.
09288.
T. H. Trinh, Y. Wu, Q. V. Le, H. He, and T. Luong. Solving olympiad geometry without human
demonstrations. Nature, 625(7995):476–482, 2024.
P. Wang, L. Li, L. Chen, F. Song, B. Lin, Y. Cao, T. Liu, and Z. Sui. Making large language models
better reasoners with alignment. arXiv preprint arXiv:2309.02144, 2023a.
P. Wang, L. Li, Z. Shao, R. Xu, D. Dai, Y. Li, D. Chen, Y. Wu, and Z. Sui. Math-shepherd: Verify
and reinforce llms step-by-step without human annotations. CoRR, abs/2312.08935, 2023b.
Z. Wang, R. Xia, and P. Liu. Generative AI for math: Part I - mathpile: A billion-token-scale
pretraining corpus for math. CoRR, abs/2312.17120, 2023c. doi: 10.48550/ARXIV.2312.17120.
URL https://doi.org/10.48550/arXiv.2312.17120.
J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. H. Chi, Q. V. Le, and D. Zhou.
Chain-of-thought prompting elicits reasoning in large language models. In NeurIPS, 2022.
URL http://papers.nips.cc/paper_files/paper/2022/hash/9d5609613524ecf
4f15af0f7b31abca4-Abstract-Conference.html.
26


T. Wei, J. Luan, W. Liu, S. Dong, and B. Wang. Cmath: Can your language model pass chinese
elementary school math test?, 2023.
M. Wenzel, L. C. Paulson, and T. Nipkow. The isabelle framework. In O. A. Mohamed, C. A.
Muñoz, and S. Tahar, editors, Theorem Proving in Higher Order Logics, 21st International
Conference, TPHOLs 2008, Montreal, Canada, August 18-21, 2008. Proceedings, volume 5170
of Lecture Notes in Computer Science, pages 33–38. Springer, 2008. doi: 10.1007/978-3-540-7
1067-7\_7. URL https://doi.org/10.1007/978-3-540-71067-7_7.
H. Xia, T. Ge, P. Wang, S.-Q. Chen, F. Wei, and Z. Sui. Speculative decoding: Exploiting
speculative execution for accelerating seq2seq generation. In H. Bouamor, J. Pino, and K. Bali,
editors, Findings of the Association for Computational Linguistics: EMNLP 2023, pages 3909–
3925, Singapore, Dec. 2023. Association for Computational Linguistics. doi: 10.18653/v1/20
23.findings-emnlp.257. URL https://aclanthology.org/2023.findings-emnlp.257.
H. Xia, Z. Yang, Q. Dong, P. Wang, Y. Li, T. Ge, T. Liu, W. Li, and Z. Sui. Unlocking efficiency
in large language model inference: A comprehensive survey of speculative decoding. arXiv
preprint arXiv:2401.07851, 2024.
S. Yao, D. Yu, J. Zhao, I. Shafran, T. L. Griffiths, Y. Cao, and K. Narasimhan. Tree of thoughts:
Deliberate problem solving with large language models. arXiv preprint arXiv:2305.10601,
2023.
L. Yu, W. Jiang, H. Shi, J. Yu, Z. Liu, Y. Zhang, J. T. Kwok, Z. Li, A. Weller, and W. Liu.
Metamath: Bootstrap your own mathematical questions for large language models. CoRR,
abs/2309.12284, 2023. doi: 10.48550/ARXIV.2309.12284. URL https://doi.org/10.485
50/arXiv.2309.12284.
Z. Yuan, H. Yuan, C. Li, G. Dong, C. Tan, and C. Zhou. Scaling relationship on learning
mathematical reasoning with large language models. arXiv preprint arXiv:2308.01825, 2023a.
Z. Yuan, H. Yuan, C. Tan, W. Wang, S. Huang, and F. Huang. Rrhf: Rank responses to align
language models with human feedback without tears. arXiv preprint arXiv:2304.05302, 2023b.
X. Yue, X. Qu, G. Zhang, Y. Fu, W. Huang, H. Sun, Y. Su, and W. Chen. Mammoth: Building
math generalist models through hybrid instruction tuning. CoRR, abs/2309.05653, 2023. doi:
10.48550/ARXIV.2309.05653. URL https://doi.org/10.48550/arXiv.2309.05653.
K. Zheng, J. M. Han, and S. Polu. Minif2f: a cross-system benchmark for formal olympiad-level
mathematics. arXiv preprint arXiv:2109.00110, 2021.
W. Zhong, R. Cui, Y. Guo, Y. Liang, S. Lu, Y. Wang, A. Saied, W. Chen, and N. Duan. AGIEval: A
human-centric benchmark for evaluating foundation models. CoRR, abs/2304.06364, 2023.
doi: 10.48550/arXiv.2304.06364. URL https://doi.org/10.48550/arXiv.2304.06364.
27


A. Appendix
A.1. Analysis of Reinforcement Learning
We provide the detailed derivation of the data source and gradient coefficient (algorithm and
reward function) across various methods, including SFT, RFT, Online RFT, DPO, PPO, and
GRPO.
A.1.1. Supervised Fine-tuning
The objective of Supervised Fine-tuning is maximizing the following objective:
J
𝑆𝐹𝑇(𝜃) = E[𝑞, 𝑜∼𝑃𝑠𝑓𝑡(𝑄, 𝑂)]
 
1
|𝑜|
|𝑜|
∑︁
𝑡=1
log 𝜋𝜃(𝑜𝑡|𝑞, 𝑜<𝑡)
!
.
(6)
The gradient of J
𝑆𝐹𝑇(𝜃) is:
∇𝜃J
𝑆𝐹𝑇= E[𝑞, 𝑜∼𝑃𝑠𝑓𝑡(𝑄, 𝑂)]
 
1
|𝑜|
|𝑜|
∑︁
𝑡=1
∇𝜃log 𝜋𝜃(𝑜𝑡|𝑞, 𝑜<𝑡)
!
.
(7)
Data Source: The dataset employed for SFT. Reward Function: This can be regarded as human
selection. Gradient Coefficient: always set to 1.
A.1.2. Rejection Sampling Fine-tuning
Rejection Sampling Fine-tuning first samples multiple outputs from the supervised fine-tuned
LLMs for each question, and then trains LLMs on the sampled outputs with the correct answer.
Formally, the objective of RFT is to maximize the following objectives:
J
𝑅𝐹𝑇(𝜃) = E[𝑞∼𝑃𝑠𝑓𝑡(𝑄), 𝑜∼𝜋𝑠𝑓𝑡(𝑂|𝑞)]
 
1
|𝑜|
|𝑜|
∑︁
𝑡=1
I(𝑜) log 𝜋𝜃(𝑜𝑡|𝑞, 𝑜<𝑡)
!
.
(8)
The gradient of J
𝑅𝐹𝑇(𝜃) is:
∇𝜃J
𝑅𝐹𝑇(𝜃) = E[𝑞∼𝑃𝑠𝑓𝑡(𝑄), 𝑜∼𝜋𝑠𝑓𝑡(𝑂|𝑞)]
 
1
|𝑜|
|𝑜|
∑︁
𝑡=1
I(𝑜)∇𝜃log 𝜋𝜃(𝑜𝑡|𝑞, 𝑜<𝑡)
!
.
(9)
Data Source: question in SFT dataset with outputs sampled from SFT model. Reward Function:
Rule (whether the answer is correct or not). Gradient Coefficient:
𝐺𝐶𝑅𝐹𝑇(𝑞, 𝑜, 𝑡) = I(𝑜) =
(
1
the answer of o is correct
0
the answer of o is incorrect
(10)
A.1.3. Online Rejection Sampling Fine-tuning
The only difference between RFT and Online RFT is that the outputs of Online RFT are sampled
from the real-time policy model 𝜋𝜃, rather than from the SFT model 𝜋𝜃𝑠𝑓𝑡. Therefore, the gradient
of online RFT is:
∇𝜃J
𝑂𝑛𝑅𝐹𝑇(𝜃) = E[𝑞∼𝑃𝑠𝑓𝑡(𝑄), 𝑜∼𝜋𝜃(𝑂|𝑞)]
 
1
|𝑜|
|𝑜|
∑︁
𝑡=1
I(𝑜)∇𝜃log 𝜋𝜃(𝑜𝑡|𝑞, 𝑜<𝑡)
!
.
(11)
28


A.1.4. Direct Preference Optimization (DPO)
The objective of DPO is:
J
𝐷𝑃𝑂(𝜃) = E[𝑞∼𝑃𝑠𝑓𝑡(𝑄), 𝑜+, 𝑜−∼𝜋𝑠𝑓𝑡(𝑂|𝑞)] log 𝜎©
­
«
𝛽1
|𝑜+|
|𝑜+|
∑︁
𝑡=1
log
𝜋𝜃(𝑜+
𝑡|𝑞, 𝑜+
<𝑡)
𝜋ref(𝑜+
𝑡|𝑞, 𝑜+
<𝑡) −𝛽1
|𝑜−|
|𝑜−|
∑︁
𝑡=1
log
𝜋𝜃(𝑜−
<𝑡|𝑞, 𝑜−
<𝑡)
𝜋ref(𝑜−
<𝑡|𝑞, 𝑜−
<𝑡)
ª
®
¬
(12)
The gradient of J
𝐷𝑃𝑂(𝜃) is:
∇𝜃J
𝐷𝑃𝑂(𝜃) = E[𝑞∼𝑃𝑠𝑓𝑡(𝑄), 𝑜+, 𝑜−∼𝜋𝑠𝑓𝑡(𝑂|𝑞)] ©
­
«
1
|𝑜+|
|𝑜+|
∑︁
𝑡=1
𝐺𝐶𝐷𝑃𝑂(𝑞, 𝑜, 𝑡)∇𝜃log 𝜋𝜃(𝑜+
𝑡|𝑞, 𝑜+
<𝑡)
−
1
|𝑜−|
|𝑜−|
∑︁
𝑡=1
𝐺𝐶𝐷𝑃𝑂(𝑞, 𝑜, 𝑡)∇𝜃log 𝜋𝜃(𝑜−
𝑡|𝑞, 𝑜−
<𝑡)ª
®
¬
(13)
Data Source: question in SFT dataset with outputs sampled from SFT model. Reward Function:
human preference in the general domain (can be ‘Rule’ in mathematical tasks). Gradient
Coefficient:
𝐺𝐶𝐷𝑃𝑂(𝑞, 𝑜, 𝑡) = 𝜎

𝛽log
𝜋𝜃(𝑜−
𝑡|𝑞, 𝑜−
<𝑡)
𝜋ref(𝑜−
𝑡|𝑞, 𝑜−
<𝑡) −𝛽log
𝜋𝜃(𝑜+
𝑡|𝑞, 𝑜+
<𝑡)
𝜋ref(𝑜+
𝑡|𝑞, 𝑜+
<𝑡)

(14)
A.1.5. Proximal Policy Optimization (PPO)
The objective of PPO is:
J
𝑃𝑃𝑂(𝜃) = E[𝑞∼𝑃𝑠𝑓𝑡(𝑄), 𝑜∼𝜋𝜃𝑜𝑙𝑑(𝑂|𝑞)] 1
|𝑜|
|𝑜|
∑︁
𝑡=1
min
 𝜋𝜃(𝑜𝑡|𝑞, 𝑜<𝑡)
𝜋𝜃𝑜𝑙𝑑(𝑜𝑡|𝑞, 𝑜<𝑡) 𝐴𝑡, clip
 𝜋𝜃(𝑜𝑡|𝑞, 𝑜<𝑡)
𝜋𝜃𝑜𝑙𝑑(𝑜𝑡|𝑞, 𝑜<𝑡) , 1 −𝜀, 1 + 𝜀

𝐴𝑡

.
(15)
To simplify the analysis, it is assumed that the model only has a single update following each
exploration stage, thereby ensuring that 𝜋𝜃𝑜𝑙𝑑= 𝜋𝜃. In this case, we can remove the min and clip
operation:
J
𝑃𝑃𝑂(𝜃) = E[𝑞∼𝑃𝑠𝑓𝑡(𝑄), 𝑜∼𝜋𝜃𝑜𝑙𝑑(𝑂|𝑞)] 1
|𝑜|
|𝑜|
∑︁
𝑡=1
𝜋𝜃(𝑜𝑡|𝑞, 𝑜<𝑡)
𝜋𝜃𝑜𝑙𝑑(𝑜𝑡|𝑞, 𝑜<𝑡) 𝐴𝑡.
(16)
The gradient of J
𝑃𝑃𝑂(𝜃) is:
∇𝜃J
𝑃𝑃𝑂(𝜃) = E[𝑞∼𝑃𝑠𝑓𝑡(𝑄), 𝑜∼𝜋𝜃𝑜𝑙𝑑(𝑂|𝑞)] 1
|𝑜|
|𝑜|
∑︁
𝑡=1
𝐴𝑡∇𝜃log 𝜋𝜃(𝑜𝑡|𝑞, 𝑜<𝑡)
(17)
Data Source: question in SFT dataset with outputs sampled from policy model. Reward Function:
reward model. Gradient Coefficient:
𝐺𝐶𝑃𝑃𝑂(𝑞, 𝑜, 𝑡, 𝜋𝜃𝑟𝑚) = 𝐴𝑡,
(18)
where 𝐴𝑡is the advantage, which is computed by applying Generalized Advantage Estimation
(GAE) (Schulman et al., 2015), based on the rewards {𝑟≥𝑡} and a learned value function 𝑉𝜓.
A.1.6. Group Relative Policy Optimization (GRPO)
The objective of GRPO is (assume 𝜋𝜃𝑜𝑙𝑑= 𝜋𝜃for simplified analysis):
J
𝐺𝑅𝑃𝑂(𝜃) = E[𝑞∼𝑃𝑠𝑓𝑡(𝑄), {𝑜𝑖}𝐺
𝑖=1 ∼𝜋𝜃𝑜𝑙𝑑(𝑂|𝑞)]
1
𝐺
𝐺
∑︁
𝑖=1
1
|𝑜𝑖|
|𝑜𝑖|
∑︁
𝑡=1
 𝜋𝜃(𝑜𝑖,𝑡|𝑞, 𝑜𝑖,<𝑡)
𝜋𝜃𝑜𝑙𝑑(𝑜𝑖,𝑡|𝑞, 𝑜𝑖,<𝑡)
ˆ
𝐴𝑖,𝑡−𝛽(
𝜋𝑟𝑒𝑓(𝑜𝑖,𝑡|𝑞, 𝑜𝑖,<𝑡)
𝜋𝜃(𝑜𝑖,𝑡|𝑞, 𝑜𝑖,<𝑡) −log
𝜋𝑟𝑒𝑓(𝑜𝑖,𝑡|𝑞, 𝑜𝑖,<𝑡)
𝜋𝜃(𝑜𝑖,𝑡|𝑞, 𝑜𝑖,<𝑡) −1)

.
(19)
29


The gradient of J
𝐺𝑅𝑃𝑂(𝜃) is:
∇𝜃J
𝐺𝑅𝑃𝑂(𝜃) = E[𝑞∼𝑃𝑠𝑓𝑡(𝑄), {𝑜𝑖}𝐺
𝑖=1 ∼𝜋𝜃𝑜𝑙𝑑(𝑂|𝑞)]
1
𝐺
𝐺
∑︁
𝑖=1
1
|𝑜𝑖|
|𝑜𝑖|
∑︁
𝑡=1

ˆ
𝐴𝑖,𝑡+ 𝛽
 𝜋𝑟𝑒𝑓(𝑜𝑖,𝑡|𝑜𝑖,<𝑡)
𝜋𝜃(𝑜𝑖,𝑡|𝑜𝑖,<𝑡) −1

∇𝜃log 𝜋𝜃(𝑜𝑖,𝑡|𝑞, 𝑜𝑖,<𝑡).
(20)
Data Source: question in SFT dataset with outputs sampled from policy model. Reward Function:
reward model. Gradient Coefficient:
𝐺𝐶𝐺𝑅𝑃𝑂(𝑞, 𝑜, 𝑡, 𝜋𝜃𝑟𝑚) = ˆ
𝐴𝑖,𝑡+ 𝛽
 𝜋𝑟𝑒𝑓(𝑜𝑖,𝑡|𝑜𝑖,<𝑡)
𝜋𝜃(𝑜𝑖,𝑡|𝑜𝑖,<𝑡) −1

,
(21)
where ˆ
𝐴𝑖,𝑡is computed based on the group reward scores.
30


MAmmoTH2: Scaling Instructions from the Web
♢Xiang Yue∗
, ♠Tuney Zheng∗
, ♠Ge Zhang∗
, ♠Wenhu Chen∗
♢Carnegie Mellon University, ♠University of Waterloo
xyue2@andrew.cmu.edu
wenhuchen@uwaterloo.ca
https://tiger-ai-lab.github.io/MAmmoTH2/
Abstract
Instruction tuning improves the reasoning abilities of large language models
(LLMs), with data quality and scalability being the crucial factors. Most instruction
tuning data come from human crowd-sourcing or GPT-4 distillation. We propose a
paradigm to efficiently harvest 10 million naturally existing instruction data from
the pre-training web corpus to enhance LLM reasoning. Our approach involves
(1) recalling relevant documents, (2) extracting instruction-response pairs, and (3)
refining the extracted pairs using open-source LLMs. Fine-tuning base LLMs on
this dataset, we build MAmmoTH2 models, which significantly boost performance on
reasoning benchmarks. Notably, MAmmoTH2-7B’s (Mistral) performance increases
from 11% to 36.7% on MATH and from 36% to 68.4% on GSM8K without training
on any in-domain data. Further training MAmmoTH2 on public instruction tuning
datasets yields MAmmoTH2-Plus, achieving state-of-the-art performance on several
reasoning and chatbot benchmarks. Our work demonstrates how to harvest large-
scale, high-quality instruction data without costly human annotation or GPT-4
distillation, providing a new paradigm for building better instruction tuning data.
25
22
72
32
61
57
35
48
85
36
73
75
29
45
85
37
65
63
34
47
86
38
72
74
20
30
40
50
60
70
80
90
100
TheoremQA
MATH
GSM8K
GPQA
MMLU-STEM
BBH
Mixtral-8x7B-Instruct
Qwen-1.5-110B
MAmmoTH2-7B-Plus (Ours)
MAmmoTH2-8x7B-Plus (Ours)
60
24
23
68.7
33.8
32.6
0
20
40
60
80
MBPP
AlpacaEval2 ArenaHard
Mixtral-8x7B-Instruct
MAmmoTH2-8x7B-Plus (Ours)
Reasoning Benchmarks
Additional Benchmarks
Figure 1: Overview of MAmmoTH2-Plus results. The MAmmoTH2-8x7B-Plus variant outperforms
Mixtral-Instruct on reasoning benchmarks, matching Qwen-1.5-110B with only 13B active parameters.
It also surpasses Mixtral-Instruct by around 10 points on general code and chatbot benchmarks.
1
Introduction
Reasoning is a fundamental aspect of human cognition and problem-solving (Clark et al., 2018;
Hendrycks et al., 2021a; Cobbe et al., 2021; Rein et al., 2023; Yue et al., 2023a). Proficiency in
∗All of the authors are core contributors to the project.
arXiv:2405.03548v4  [cs.CL]  23 May 2024


Extract
Web
Recall
WebInstruct: 10M instruction data from the web
Distillation from GPT-4
Human annotation
Seed Data 
Costly, usually small-scale
Synthetic
Annotated
Prone to bias & hallucinations
Diverse, high quality and large-scale
Raw Doc
Extracted QA
Annotators
Previous Methods
Our Method
Refine
Refined QA
Figure 2: Comparison between our dataset curation method and previous studies.
reasoning is essential for advancing scientific knowledge, developing new technologies, and making
informed decisions in various contexts. Recently, large language models (LLMs) (Brown et al., 2020;
Ouyang et al., 2022; Touvron et al., 2023a,b; Achiam et al., 2023; Team et al., 2023) have shown
remarkable progress in various NLP tasks. However, their ability to perform complex reasoning
tasks (Lin et al., 2024) in the domains of mathematics, science, and engineering is still limited.
Recent studies have extensively explored how to enhance base LLMs’ reasoning abilities. The two
main approaches are continued training and instruction tuning. Continued training trains LLMs on
large-scale filtered documents (Lewkowycz et al., 2022; Taylor et al., 2022; Azerbayev et al., 2023;
Shao et al., 2024; Ying et al., 2024). Instruction tuning seeks to employ supervised fine-tuning loss on,
usually small-scale, high-quality instruction-response pairs (Ouyang et al., 2022; Chung et al., 2024).
While human-annotated instruction datasets (Cobbe et al., 2021; Hendrycks et al., 2021b; Amini
et al., 2019) are often limited in scale, recent studies (Yu et al., 2023; Yue et al., 2023b; Toshniwal
et al., 2024; Li et al., 2024a; Tang et al., 2024) attempt to prompt GPT-4 with seed data to increase
the scalability. However, the synthesized instruction data becomes highly biased, not diverse, and
prone to a high degree of hallucination.
To address these limitations, we propose to discover naturally existing instruction data from the
web (Figure 2). We argue that the pre-training corpus (e.g., Common Crawl) already contains a vast
amount of high-quality instruction data for LLM reasoning. For example, the web corpus contains a
large amount of educational materials in the form of instruction-following pairs. These documents
range across various domains like math, science, engineering, and humanities. Such readily available
instruction data is not only diverse but also of high quality. However, such instruction data is highly
dispersed across the corpus, which makes it particularly challenging to discover.
In this paper, we aim to mine these instruction-response pairs from the web using a three-step pipeline.
(1) Recall step: We create a diverse seed dataset by crawling several quiz websites. We use this
seed data to train a fastText model (Joulin et al., 2016) and employ it to recall documents from
Common Crawl (Computer, 2023). GPT-4 is used to trim down the recalled documents by their root
URL. We obtain 18M documents through this step. (2) Extract step: We utilize open-source LLMs
like Mixtral (Jiang et al., 2024) to extract Q-A pairs from these documents, producing roughly 5M
candidate Q-A pairs. (3) Refine step: After extraction, we further employ Mixtral-8×7B (Jiang et al.,
2024) and Qwen-72B (Bai et al., 2023) to refine (Zheng et al., 2024b) these candidate Q-A pairs. This
refinement operation aims to remove unrelated content, fix formality, and add missing explanations
to the candidate Q-A pairs. This refinement operation is pivotal to maintaining the quality of the
mined Q-A pairs. Eventually, we harvest a total of 10M instruction-response pairs through these
steps. Unlike existing instruction-tuning dataset, our dataset WEBINSTRUCT is purely mined from
the Web without any human crowdsourcing or GPT-4 distillation.
We validate the effectiveness of WEBINSTRUCT by training MAmmoTH2 on various base models
(Figure 1), including Mistral-7B (Jiang et al., 2023), Llama3-8B (Meta, 2024), Mixtral-8×7B (Jiang
et al., 2024), and Yi-34B (Young et al., 2024). MAmmoTH2 significantly outperforms the base models
on seven held-out reasoning benchmarks: TheoremQA (Chen et al., 2023b), GSM8K (Cobbe et al.,
2021), MATH (Hendrycks et al., 2021b), ARC-C (Clark et al., 2018), MMLU-STEM (Hendrycks
et al., 2021b), GPQA (Rein et al., 2023), and BBH (Suzgun et al., 2022). MAmmoTH2-7B improves
Mistral-7B’s performance by an average of 14 absolute points, while MAmmoTH2-34B enhances Yi-
34B’s performance by an average of 5.8 absolute points. Notably, Mistral-7B’s MATH accuracy can
rise from 11.2% to 36.7% after training on WEBINSTRUCT. As our dataset contains no in-domain
data from our evaluation benchmarks, this highlights the models’ strong generalization ability.
We further enhance MAmmoTH2’s performance on code generation, math reasoning, and instruction-
following tasks by tuning it on open-source instruction datasets, including OpenHermes2.5 (Teknium,
2


2023), Code-Feedback (Zheng et al., 2024c), and Math-plus. The resulting model, MAmmoTH2-Plus,
excels on seven reasoning benchmarks and other general tasks.
MAmmoTH2-7B-Plus and
MAmmoTH2-8B-Plus achieve state-of-the-art performance on TheoremQA, ARC-C, MMLU-STEM,
GPQA, and BBH, and competitive results on MATH (45%) and GSM8K (85%). MAmmoTH2-Plus
also performs well on general tasks, with MAmmoTH2-7B-Plus showing promising results on Hu-
manEval and MBPP, and MAmmoTH2-8×7B leading the AlpacaEval 2.0 and Arena Hard leaderboards.
Interestingly, MAmmoTH2-8B-Plus and Llama-3-8B-Instruct, both tuned from Llama-3-base using
datasets of the same size (10M), provide an apple-to-apple comparison. The only distinction is that
Llama-3-8B-Instruct is trained on 10M human-annotated dataset while we do not require any human
annotation. MAmmoTH2-8B-Plus outperforms Llama-3-Instruct by 6 points on reasoning tasks while
matching its performance on general tasks, reflecting WEBINSTRUCT’s cost-effectiveness advantage.
MAmmoTH2-Plus consistently surpasses official instruction models like Mixtral-Instruct on chat
benchmarks. These results demonstrate the effectiveness of our approach to scale up instruction data
from the web and offer a new perspective for future instruction tuning studies.
2
WEBINSTRUCT
In this section, we outline the process of constructing WEBINSTRUCT. Specifically, we divide the
data collection pipeline into three stages: (1) relevant document recall from the web corpus, (2) Q-A
pair extraction from recalled document, and (3) Q-A pair refinement. The full pipeline is depicted
in Figure 3 and an example for extraction and refinement is provided in Figure 4.
2.1
Recall from Common Crawl
In contrast to previous math-centric approaches (Paster et al., 2023; Wang et al., 2023c; Shao et al.,
2024), we aim for broad coverage of disciplines such as math, science, engineering, etc. Therefore,
careful balancing of the seed data is necessary to ensure diversity. However, publicly available
training datasets are mostly limited to mathematics. To address this issue, we propose to crawl new
Fast-Text
LLM Extraction
LLM Refinement
1
2
Recall from Web
Q-A Pair Extraction
Pre-process
Fast-Text
CC
CC
Seed
Decontaminate
URL Filter
3 Q-A Pair Refinement
URL Select
Figure 3: Step 1: Recall relevant documents from Common Crawl. Step 2: Extracting Q-A pairs.
Step 3: Refine with the extracted Q-A pairs.
Topics Science\nAnatomy&Physiology\nAstronomy\nAstrophysics
\nBiology\nChemistry \n...Socratic Meta...Featured Answers
How do you simplify #((u^4v^3)/(u^2v^-1)^4)^0# and write it using only 
positive exponents?
Answer by NickTheTurtle (Apr 1, 2017)
Explanation:\nAnything raised to the 0^(th) power is simply 1.
\n\nRelated Questions\n What is the quotient of powers property?
\n\n\n\n How do you simplify expressions using the quotient
rule?…\nImpact of this question\n1274 views around the world
#Apps\niOS\nAndroid\nLinks\n[Privacy](#)\n[Terms](#)\n[Help](#)
Raw Docs
Question: How do you simplify (𝑢! 𝑣"
𝑢# 𝑣$% !
⁄
)& and write it using
only positive exponents?
Answer: Explanation: Anything to the 0𝑡ℎpower is just simply 1.
Extracted QA
Question:
How do you simplify (𝑢! 𝑣"
𝑢# 𝑣$% !
⁄
)&and write it using only
positive exponents?
Answer:
To simplify the expression (𝑢! 𝑣"
𝑢# 𝑣$% !
⁄
)& and rewrite it
using only positive exponents, we start by evaluating the
expression from the innermost operation outward.
First, consider the exponent of zero on the entire fraction:
Any expression raised to the power of zero is equal to 1 (provided
the base is not zero). Therefore, (𝑢! 𝑣"
𝑢# 𝑣$% !
⁄
)& = 1
This simplification makes the other calculations unnecessary
because raising any non-zero expression to the power of zero will
always result in 1.
So, the simplified expression is simply: 1
Rewritten QA
Unformatted Text, Site Information, Ads
Formatted QA but lacking detailed solutions
Formatted QA augmented with detailed solutions
Figure 4: An illustrating example from WEBINSTRUCT for the extraction and refinement step.
3


exam problems from several educational websites. These sites contain diverse problems from various
disciplines, helping to ensure diversity. We crawled 100K seed data as positive training examples
and randomly selected 100K negative documents from CC (Computer, 2023) to train a fastText
model (Joulin et al., 2016). The trained fastText model is used to recall relevant documents. We
employ the open-source fastText library with a vector dimension of 256 to train the model for 3
epochs, with a learning rate of 0.1, a maximum n-gram length of 3, and a maximum number of
word occurrences of 3. We recalled 100B tokens using the trained fasttext model from an internal
CC. These raw web documents are further grouped by their domains (root URL) and only domains
with more than 1000 documents are retained. We extracted roughly 600K domains from the recalled
documents. We then prompt GPT-3.5 to scan through the domains and automatically select those
that might contain instruction data. Around 50K domains are further labeled as positive samples
by GPT-3.5. Note that all the recalled documents in the first round are not kept for further usage
in Q-A Pair Extraction and Refinement. Next, we sample documents from the selected domains as
positive examples, and documents from the non-selected domains and general Common Crawl as
negative examples to re-train an improved fastText classifier. The newly trained fastText classifier
is used to recall documents. We recalled 40B tokens using the newly trained fastText model. We
prompt GPT-4 to sift through the recalled domains again, ultimately leading to 18M raw documents,
primarily originating from the desired websites.
2.2
Q-A Pair Extraction
We observe that a significant number of naturally existing Q-A pairs are present in the 18M documents.
However, these Q-A pairs are interspersed with a high volume of noise such as ads, markups,
boilerplate, etc. Our preliminary training on these raw documents only yields limited gains.
First, we carefully pre-process the HTML to pre-extract useful content from the recalled documents.
This is mostly rule-based filtering to clean site information, ads, HTML boilerplate, etc. This step
significantly reduces the document length for the next stage. We then prompt Qwen-72B (Bai et al.,
2023) to identify the question and answer pairs from the preprocessed documents. Specifically, we
provide a few in-context examples to help the model understand what to extract. We also allow the
model to return void if no natural question-answer pairs exist. In this stage, only 30% of the recalled
documents were identified as containing naturally existing Q-A pairs, resulting in roughly 5M Q-A
pairs as our candidates for the next step. However, these candidates still contain a substantial amount
of unrelated content and formality issues. Besides that, a large portion of the extracted Q-A pairs also
lack explanations for how the answer is derived. Therefore, we propose to perform another round of
refinement to increase the data quality.
To avoid contamination, we follow previous work (Shao et al., 2024) and filter out web pages
containing questions or answers to all of our evaluation benchmarks. Specifically, we filter out all
web pages that contain n-grams (n = 10) string matches with either the questions or answers.
2.3
Q-A Pair Refinement
To further improve the extracted Q-A pair candidates, we propose refining them using LLMs. In this
step, we prompt Mixtral-22B×8 (Jiang et al., 2024) and Qwen-72B (Bai et al., 2023) to reformat
the extracted Q-A pairs. If the answer does not contain any explanation, these two LLMs will
attempt to complete the intermediate reasoning steps leading to the given answer. We adopt two
models to increase the diversity of our dataset. Eventually, we harvest 10M Q-A pairs as our final
instruction-tuning dataset WEBINSTRUCT.
2.4
Dataset Statistics
To better distinguish our dataset from the existing ones, we include a summarization table in Table 1.
It can be observed that most SFT datasets contain less than 1M samples but are of high quality.
XwinMath (Li et al., 2024a) is the largest dataset, scaling up to over 1M samples through GPT4
synthesis, while OpenMathInstruct (Toshniwal et al., 2024) has not been generated using GPT-4 but
instead uses Mixtral-8x7B Jiang et al. (2024). However, the seed data for both datasets is only based
on GSM and MATH, leading to narrow domain coverage. In contrast, continue-training (CT) datasets
are normally filtered from the web with much larger size, often exceeding 10B tokens and even rising
to 120B tokens. However, continued pre-training on these massive datasets can be not only expensive
4


Table 1: The list of existing supervise-fine-tuning (SFT) and continue-training (CT) datasets. SFT
datasets are primarily from academic NLP sources or synthesized by GPT-3.5/4 using seed data. CT
datasets are larger but nosier. Our dataset falls between these two types.
Dataset
#Pairs
Domain
Format
Dataset Source
FLAN V2 (Chung et al., 2024)
100K
General
SFT
NLP data + Human CoT
Self-Instruct (Wang et al., 2023b)
82K
General
SFT
Generated by GPT3
GPT4-Alpaca (Taori et al., 2023)
52K
General
SFT
Generated by GPT4
SuperNI (Wang et al., 2022)
96K
General
SFT
NLP Datasets
Tora (Gou et al., 2023)
16K
Math
SFT
GSM+MATH Synthesis by GPT4
WizardMath (Luo et al., 2023)
96K
Math
SFT
GSM+MATH Synthesis by GPT4
MathInstruct (Yue et al., 2023b)
262K
Math
SFT
Math datasets Synthesis by GPT4
MetaMathQA (Yu et al., 2023)
395K
Math
SFT
GSM+MATH Synthesis by GPT3.5
XwinMath (Li et al., 2024a)
1.4M
Math
SFT
GSM+MATH Synthesis by GPT4
OpenMathInstruct (Toshniwal et al., 2024)
1.8M
Math
SFT
GSM+MATH Synthesis by Mixtral
Dataset
#Tokens
Domain
Format
Dataset Source
OpenWebMath (Paster et al., 2023)
12B
Math
LM
Filtered from Web
MathPile (Wang et al., 2023c)
10B
Math
LM
Filtered from Web
Cosmopeida (Ben Allal et al., 2024)
25B
General
LM
Synthesized by Mixtral
MINERVA (Lewkowycz et al., 2022)
38B
Math
LM
Filtered from Web
Proof-Pile-2 (Azerbayev et al., 2023)
55B
Math
LM
OpenWebMath+Arxiv+Code
Galactica (Taylor et al., 2022)
106B
Math & Sci.
LM
Filtered from Web
DeepseekMath (Shao et al., 2024)
120B
Math
LM
Recalled from Web
WEBINSTRUCT
(10M) 5B
Math & Sci.
SFT
Recall and Extracted from Web
but also ineffective due to the high noise ratio. WEBINSTRUCT, with roughly 5B tokens, strikes a
good balance between scalability and quality. It approaches the scalability of common CT datasets
while maintaining high quality through the three-step construction pipeline. This makes our dataset
unique compared to other alternatives.
2.5
Additional Public Instruction Datasets
To further enhance the diversity and quality of our dataset, we fine-tune MAmmoTH2 on several open-
source instruction tuning datasets. These datasets are carefully selected based on their relevance to
different reasoning subjects. Additionally, we consider some chat datasets to balance reasoning ability
and general chat ability. The open-source datasets we incorporate are OpenHermes 2.5 (Teknium,
2023), Code-Feedback (Zheng et al., 2024c) and our Math-Plus, which is an augmented version of
MetaMathQA (395K) (Yu et al., 2023) and Orca-Math (200K) (Mitra et al., 2024). More details of
the public datasets can be found in Appendix A.
3
Experimental Setup
3.1
Training Setup
We unify all the samples in our instruction dataset to conform to the structure of a multi-turn instruction
tuning dataset. This standardization ensures that the fine-tuned models can process data consistently,
regardless of the original dataset formats. We select the open-source models Mistral 7B (Jiang et al.,
2023), Mixtral 8×7B (Jiang et al., 2024), Llama-3 8B (Meta, 2024), and Yi-34B (Young et al., 2024)
as our base models. We fine-tune these models to validate our WEBINSTRUCT at multiple scales
using the LLaMA-Factory (Zheng et al., 2024d) library. We use a learning rate of 5e-6 for Mistral 7B
and 1e-5 for Mixtral, Llama-3 8B, and Yi 34B. The global batch size is set to 512 with a maximum
sequence length of 4096. We employ a cosine scheduler with a 3% warm-up period for 2 epochs. To
efficiently train the models, we utilize DeepSpeed (Rasley et al., 2020) with the ZeRO-3 stage. All
the models are trained with 32 A100 GPUs.
3.2
Evaluation Datasets
To rigorously assess the capabilities of models in reasoning abilities across different domains, we
utilize several widely used datasets, GSM8K (Cobbe et al., 2021), MATH (Hendrycks et al., 2021b),
TheoremQA (Chen et al., 2023b), BIG-Bench Hard (BBH) (Suzgun et al., 2022), ARC-C (Clark
5


Table 2: Main results on reasoning datasets. Models without the ’-Instruct’ suffix refer to the released
base models. Results are taken from official papers or blogs when available; otherwise, we use our
own evaluation script. Underscored results represent the best baseline scores under the size constraint.
All models are inferred with few-shot CoT: TheoremQA (5-shot), MATH (4-shot), GSM8K (4-shot),
GPQA (5-shot), MMLU-STEM (5-shot), BBH (3-shot), and ARC-C (8-shot).
Model
TheoremQA
MATH
GSM8K
GPQA
MMLU-ST
BBH
ARC-C
AVG
GPT-4-Turbo-0409
48.4
69.2
94.5
46.2
76.5
86.7
93.6
73.6
Parameter Size between 20B and 110B
Qwen-1.5-110B
34.9
49.6
85.4
35.9
73.4
74.8
91.6
63.6
Qwen-1.5-72B
29.3
46.8
77.6
36.3
68.5
68.0
92.2
59.8
Deepseek-LM-67B
25.3
15.9
66.5
31.8
57.4
71.7
86.8
50.7
Yi-34B
23.2
15.9
67.9
29.7
62.6
66.4
89.5
50.7
Llemma-34B
21.1
25.0
71.9
29.2
54.7
48.4
69.5
45.7
Mixtral-8×7B
23.2
28.4
74.4
29.7
59.7
66.8
84.7
52.4
Mixtral-8×7B-Instruct
25.3
22.1
71.7
32.4
61.4
57.3
84.7
50.7
Intern-Math-20B
17.1
37.7
82.9
28.9
50.1
39.3
68.6
46.4
Trained only with WEBINSTRUCT (All evaluations are held-out)
MAmmoTH2-34B
30.4
35.0
75.6
31.8
64.5
68.0
90.0
56.4
∆over Yi
+7.2
+19.1
+7.7
+2.1
+2.9
+1.2
+0.5
+5.8
MAmmoTH2-8x7B
32.2
39.0
75.4
36.8
67.4
71.1
87.5
58.9
∆over Mixtral
+9.2
+10.6
+1.0
+7.1
+7.4
+3.3
+2.8
+6.5
Continue trained with additional instruction datasets (All held-out except MATH and GSM8K)
MAmmoTH2-8x7B-Plus
34.1
47.0
86.4
37.8
72.4
74.1
88.4
62.9
∆over Qwen-1.5-110B
-0.8
-2.6
+1.0
+1.5
-1.0
-0.7
-4.0
-0.7
Parameter Size = 7B or 8B
Deepseek-7B
15.7
6.4
17.4
25.7
43.1
42.8
47.8
28.4
Qwen-1.5-7B
14.2
13.3
54.1
26.7
45.4
45.2
75.6
39.2
Mistral-7B
19.2
11.2
36.2
24.7
50.1
55.7
74.2
38.8
Gemma-7B
21.5
24.3
46.4
25.7
53.3
57.4
72.5
43.0
Llemma-7B
17.2
18.0
36.4
23.2
45.2
44.9
50.5
33.6
WizardMath-7B-1.1
11.7
33.0
83.2
28.7
52.7
56.7
76.9
49.0
Abel-7B-002
19.3
29.5
83.2
30.3
29.7
32.7
72.5
42.5
Intern-Math-7B
13.2
34.6
78.1
22.7
41.1
48.1
59.8
42.5
Rho-1-Math-7B
21.0
31.0
66.9
29.2
53.1
57.7
72.7
47.3
Deepseek-Math-7B
25.3
34.0
64.2
29.2
56.4
59.5
67.8
48.0
Deepseek-Math-Instruct
23.7
44.3
82.9
31.8
59.3
55.4
70.1
52.5
Llama-3-8B
20.1
21.3
54.8
27.2
55.6
61.1
78.6
45.5
Llama-3-8B-Instruct
22.8
30.0
79.5
34.5
60.2
66.0
80.8
53.4
Trained only with WEBINSTRUCT (All evaluations are held-out)
MAmmoTH2-7B
29.0
36.7
68.4
32.4
62.4
58.6
81.7
52.8
∆over Mistral
+9.8
+25.5
+32.2
+7.7
+12.3
+2.9
+7.5
+14.0
MAmmoTH2-8B
32.2
35.8
70.4
35.2
64.2
62.1
82.2
54.3
∆over Llama3
+12.2
+14.5
+15.6
+8.0
+8.6
+1.0
+3.6
+8.8
Continue trained with additional instruction datasets (All held-out except MATH and GSM8K)
MAmmoTH2-7B-Plus
29.2
45.0
84.7
36.8
64.5
63.1
83.0
58.0
MAmmoTH2-8B-Plus
32.5
42.8
84.1
37.3
65.7
67.8
83.4
59.1
∆over best baseline
+7.2
+0.7
+1.5
+2.8
+5.5
+1.8
+2.6
+5.7
et al., 2018), GPQA (Rein et al., 2023), MMLU-STEM (Hendrycks et al., 2021a). These datasets
collectively enable a comprehensive assessment of language models’ reasoning prowess across a
spectrum of complexity and realism. The details of the evaluation datasets are in Appendix B.
We further evaluate the models on additional code generation tasks (including HumanEval (Chen
et al., 2021), MBPP (Austin et al., 2021) and their augmented version (Liu et al., 2024)), general
LLM benchmarks like MMLU (Hendrycks et al., 2021a) and its recent robust and challenging version
MMLU-Pro (TIGER-Lab, 2024). We also consider chat benchmarks like MT-Bench (Zheng et al.,
2024a), AlpacaEval 2.0 (Li et al., 2023), and Arena Hard (Li et al., 2024b) to demonstrate the
generalizability of WEBINSTRUCT and WEBINSTRUCT-PLUS on more general LLM benchmarks.
6


4
Main Results
4.1
Experimental Results on Reasoning Benchmarks
Table 2 presents our main results, with existing models partitioned into two tracks based on their
parameter size. For 7B parameter models, we observe that our model trained solely with WEBIN-
STRUCT achieves significant improvements over the base models. For instance, MAmmoTH2-7B boosts
the performance of Mistral-7B by an average of 14 points. Notably, WEBINSTRUCT does not contain
any training data from these evaluation benchmarks, making all evaluations essentially held-out. The
substantial performance gains demonstrate the strong generalization capabilities of MAmmoTH2-7B.
Similarly, MAmmoTH2-8B boosts the performance of Llama-3-8B-base by an average of 8.8 points.
We also experiment with larger models like Yi-34B and Mixtral to show that the performance gains
are consistent across the board. Notably, Yi-34B’s performance on MATH also increases by 19%
after training on WEBINSTRUCT.
Further tuning on several additional public datasets also significantly enhances performance. The
MAmmoTH2-Plus model family achieves state-of-the-art results across the board. For example,
MAmmoTH2-Plus’s performance on TheoremQA, GPQA, and ARC-C represents the best-known
results for any model under 10B parameters. MAmmoTH2-7B-Plus’s performance on MATH and
GSM is also close to the best-known results. We also show the results of the models solely trained on
the additional public datasets in Appendix E.
An interesting comparison is between MAmmoTH2-8B-Plus and Llama3-Instruct, as both models
are trained from the Llama3-base. Llama-3-instruct was trained on a 10M human-annotated in-
struction dataset along with public datasets, similar to WEBINSTRUCT combined with additional
public datasets. Therefore, these two models are highly comparable. Our experiments show that
MAmmoTH2-8B-Plus outperforms Llama3-Instruct by an average of 6% across the benchmarks. This
substantial gain indicates that WEBINSTRUCT is highly cost-effective. For larger models, we found
that MAmmoTH2-8x7B-Plus can even match the performance of Qwen-1.5-110B with only 13B active
parameters. These results demonstrate the effectiveness of our scalable instruction tuning approach.
4.2
Additional Experimental Results
To further demonstrate the capabilities of our models beyond the reasoning benchmarks presented
in Table 2, we conduct additional experiments to evaluate their performance on code generation,
general language understanding, and instruction-following tasks. Table 3 showcases the results of var-
ious models on code generation tasks. The MAmmoTH2-7B-Plus model exhibits strong performance,
achieving the highest average scores of 66.1 and 58.2 on HumanEval(+) and MBPP(+) datasets,
respectively. It outperforms the official instruct counterparts like Mistral-7B-Instruct-v0.2 on these
metrics, indicating its superior code generation abilities.
To assess the general language understanding and instruction-following capabilities of our models, we
evaluate them on a range of benchmarks, as shown in Table 3. The MAmmoTH2-Plus models exhibit
strong performance across these tasks, showcasing their versatility and robustness. For example,
MAmmoTH2-8×7B-Plus achieves the highest scores on AlpacaEval 2.0 and Arena Hard leaderboards,
surpassing competitive models like GPT-3.5-Turbo and Tulu-2-DPO-70B (Ivison et al., 2023).
2M
4M
6M
8M 10M
# of Instructions
20
25
30
Accuracy (%)
MATH
2M
4M
6M
8M 10M
# of Instructions
15
20
25
Accuracy (%)
TheoremQA
2M
4M
6M
8M 10M
# of Instructions
72.5
75.0
77.5
80.0
Accuracy (%)
ARC-C
Extracted QA (LM Loss)
Refined QA (LM Loss)
Refined QA (SFT Loss)
Figure 5: Mistral-7B model reasoning performance improves with scaling instructions. Additionally,
SFT Loss is a more effective learning approach compared to LM Loss.
7


Table 3: Evaluation of code generation, instruction-following and MMLU(-Pro) performance for
various models. We report the average of HumanEval(+) and MBPP (+) accuracy as the code
generation performance (breakdown results are in Appendix D). Baseline scores are sourced
from the original papers or the EvalPlus, MT-Bench, AlpacaEval 2.0, Arena Hard and MMLU-Pro
leaderboards. (“-”) indicates that the score is not available from the sources. MAmmoTH2-Plus
exhibits strong general conversational ability and excels at multitask language understanding across a
wide range of domains compared to their official instruct counterparts and larger models.
Code
Generation
MT-Bench
Alpaca
Eval 2.0
Arena
Hard
MMLU
MMLU-Pro
GPT-4-1106-preview
85.6 (77.5)
9.32
50.0
-
-
-
GPT-3.5-Turbo-1106
79.7 (70.2)
8.32
19.3
18.9
-
-
GPT-3.5-Turbo-0301
-
7.94
18.1
18.1
70.0
-
Tulu-2-DPO-70B
51.2 (43.0)
7.89
21.2
15.0
67.8
40.5
Llama-2-70b-chat
31.4 (26.5)
6.86
14.7
11.6
63.0
33.6
Yi-34B-Chat
38.7 (32.6)
7.86
27.2
23.1
73.5
42.1
Mistral-7B-Instruct-v0.2
43.4 (36.5)
7.60
17.1
12.6
60.8
30.8
Llama-3-8B-Instruct
65.8 (58.0)
8.02
22.9
20.6
67.2
40.9
Mixtral-8×7B-Instruct-v0.1
52.3 (44.7)
8.30
23.7
23.4
70.6
41.0
MAmmoTH2-7B-Plus
66.1 (58.2)
7.88
23.4
14.6
63.3
40.9
MAmmoTH2-8B-Plus
61.9 (53.3)
7.95
18.5
16.6
64.6
43.4
MAmmoTH2-8x7B-Plus
63.3 (55.3)
8.20
33.8
32.6
68.3
50.4
The strong performance of MAmmoTH2 on code generation and general language understanding tasks,
as evidenced by Table 3, demonstrates that our method does not overfit to the reasoning benchmarks.
Instead, it shows the models’ ability to generalize well to a wide range of tasks, highlighting their
versatility and robustness. These additional experiments further validate the effectiveness of our
WEBINSTRUCT in developing powerful and flexible language models.
5
Ablation Study
5.1
Scaling Effect of Instructions
We first investigate the impact of model scaling and loss functions on the performance of language
models across three representative tasks: MATH, TheoremQA, and ARC-C. We train models with
varying training samples (1M to 10M) using extracted QA and refined QA data, and compare the
effectiveness of two training losses: LM loss and SFT loss. Figure 5 shows that increasing model
size and using SFT Loss with synthetic data consistently improves accuracy across all tasks. These
findings demonstrate the importance of model scaling and supervised fine-tuning with synthetic data
for enhancing language model performance in various domains.
5.2
Comparison of Two Refined Models
Table 4: Comparison of the two data-refining LLMs. We
train the three models with the same steps.
Data
GSM
MATH
MMLU-S
Theo.
ARC.
Mixtral
62.9
29.1
56.5
26.1
78.3
Qwen
65.4
28.9
60.6
23.5
80.8
Merged
65.6
31.0
60.5
24.8
81.8
To assess the effectiveness of the Q-
A pair refinement process by different
LLMs, we conducted experiments by
training three mistral-7B models: one on
the data refined by Mixtral-22B×8, an-
other on the data refined by Qwen-72B,
and a third on the merged samples re-
fined by both models. For a fair compari-
son, we trained the models with the same
9000 steps and a global batch size of 512. Our results show that the model trained on Mixtral-22B×8
refined data achieves comparable performance to the one trained on Qwen-72B refined data. The
model trained on the merged samples consistently outperforms the models trained on data refined by
individual LLMs. This demonstrates the effectiveness of using multiple LLMs for refinement, as it
leads to a more diverse and comprehensive dataset.
8


5.3
Comparison of Different Domains and Sources.
To understand how each domain (e.g., math, science, others) and data source (e.g., forum websites
and education websites) contribute to the training, we train Mistral 7B on the subsets of different
domains and data sources. Details of how we obtain domain labels can be found in Appendix G.
Table 5: Impact of different data domains and sources.
Data Source
GSM
MATH
MMLU-S
Theo.
ARC.
Base
47.4
15.7
51.4
17.3
77.6
Forum
51.0
24.0
54.7
21.0
78.2
Education
58.0
24.8
54.3
23.2
79.5
Math
52.9
27.3
51.6
21.7
74.1
Science
54.4
23.7
58.9
21.0
83.6
Other
59.4
20.8
55.3
21.1
79.4
As shown in Table 5, training on differ-
ent domains and data sources leads to
varied performance across the evaluation
benchmarks. The education website data
source consistently outperforms the fo-
rum data source, indicating the higher
quality of educational questions. Inter-
estingly, while the math domain excels
on MATH, it does not lead to significant
improvements on GSM8K, another math-
focused dataset, suggesting that training
on a single math dataset may not gener-
alize well to other math benchmarks. Furthermore, training solely on the math domain does not yield
substantial gains on science and STEM benchmarks, highlighting the need for diverse training. In
contrast, the "Other" domain, which includes a diverse range of subjects, achieves the highest score
on GSM8K, emphasizing the importance of diversity in the training data.
5.4
Case Study
Figure 6: Quality distribution of 50 sam-
pled refined QA examples.
We further conduct a case study examining the quality
of extracted and refined QA pairs from the dataset. We
showcase some good and bad cases in Appendix J. We
observe that the question/answer pairs extracted from well-
formed exam and education websites are of high quality.
The common issue is that a large portion of extracted
answers do not contain intermediate rationale (chain-of-
thought). This issue could lead to worse generalization.
Therefore, we prompt Mixtral and Qwen-72B to complete
the intermediate steps. We observe that the success rate
of such completion is relatively high. However, there are
cases where the extracted question/answer pairs contain
serious formatting issues, which pose challenges for the
following refinement step. Besides these issues, we also
observe that LLMs can sometimes modify the intention of
the originally extracted content, causing hallucinations.
To quantify the error percentages, we randomly sample 50 refined QA examples and ask the human
annotators to compare whether the refined examples are correct and significantly better than the
extracted ones in terms of format and intermediate solutions. As we can see from Figure 6, 78%
examples have been improved after refinement and only 10% examples introduce hallucinations after
refinement. Overall, our case study reveals that the harvested instruction tuning dataset is generally
accurate with a low error rate.
6
Conclusion
In this paper, we argue that the web corpus contains a vast amount of high-quality instruction data
across various domains. To mine this data, we develop a three-step pipeline consisting of recall,
extraction, and refinement steps. Through this pipeline, we harvest WEBINSTRUCT, a total of
10M diverse, high-quality instruction-response pairs and train language models. Our experiments
demonstrate that MAmmoTH2 exhibits significantly enhanced science reasoning abilities compared to
the baseline models. Our work showcases the potential of harnessing the vast amount of instruction
data in the web corpus to democratize the development of LLMs with enhanced reasoning capabilities.
9


References
J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt,
S. Altman, S. Anadkat, et al. Gpt-4 technical report. ArXiv preprint, abs/2303.08774, 2023. URL
https://arxiv.org/abs/2303.08774.
A. Amini, S. Gabriel, S. Lin, R. Koncel-Kedziorski, Y. Choi, and H. Hajishirzi. MathQA: Towards
interpretable math word problem solving with operation-based formalisms. In Proceedings of the
2019 Conference of the North American Chapter of the Association for Computational Linguistics:
Human Language Technologies, Volume 1 (Long and Short Papers), pages 2357–2367, 2019. doi:
10.18653/v1/N19-1245. URL https://aclanthology.org/N19-1245.
J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry,
Q. Le, et al. Program synthesis with large language models. ArXiv preprint, abs/2108.07732, 2021.
URL https://arxiv.org/abs/2108.07732.
Z. Azerbayev, H. Schoelkopf, K. Paster, M. Dos Santos, S. M. McAleer, A. Q. Jiang, J. Deng,
S. Biderman, and S. Welleck. Llemma: An open language model for mathematics. In The Twelfth
International Conference on Learning Representations, 2023. URL https://openreview.net/
forum?id=4WnqRR915j.
J. Bai, S. Bai, Y. Chu, Z. Cui, K. Dang, X. Deng, Y. Fan, W. Ge, Y. Han, F. Huang, et al. Qwen
technical report. ArXiv preprint, abs/2309.16609, 2023. URL https://arxiv.org/abs/2309.
16609.
L. Ben Allal, A. Lozhkov, G. Penedo, T. Wolf, and L. von Werra. Cosmopedia, 2024. URL
https://huggingface.co/datasets/HuggingFaceTB/cosmopedia.
T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam,
G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh,
D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark,
C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Language models are few-shot
learners. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in
Neural Information Processing Systems 33: Annual Conference on Neural Information Processing
Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL https://proceedings.
neurips.cc/paper/2020/hash/1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html.
M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y. Burda,
N. Joseph, G. Brockman, et al.
Evaluating large language models trained on code.
ArXiv
preprint, abs/2107.03374, 2021. URL https://arxiv.org/abs/2107.03374.
W. Chen, X. Ma, X. Wang, and W. W. Cohen. Program of thoughts prompting: Disentangling
computation from reasoning for numerical reasoning tasks. Transactions on Machine Learning
Research, 2023a. URL https://openreview.net/forum?id=YfZ4ZPt8zd.
W. Chen, M. Yin, M. Ku, P. Lu, Y. Wan, X. Ma, J. Xu, X. Wang, and T. Xia. Theoremqa: A theorem-
driven question answering dataset. In The 2023 Conference on Empirical Methods in Natural
Language Processing, 2023b. URL https://aclanthology.org/2023.emnlp-main.489/.
H. W. Chung, L. Hou, S. Longpre, B. Zoph, Y. Tay, W. Fedus, Y. Li, X. Wang, M. Dehghani,
S. Brahma, et al. Scaling instruction-finetuned language models. Journal of Machine Learning
Research, 25(70):1–53, 2024. URL https://arxiv.org/pdf/2210.11416.
P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord. Think you have
solved question answering? try arc, the ai2 reasoning challenge. ArXiv preprint, abs/1803.05457,
2018. URL https://arxiv.org/abs/1803.05457.
K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton,
R. Nakano, et al. Training verifiers to solve math word problems. ArXiv preprint, abs/2110.14168,
2021. URL https://arxiv.org/abs/2110.14168.
T. Computer. Redpajama: an open dataset for training large language models, 2023. URL https:
//github.com/togethercomputer/RedPajama-Data.
10


L. Gao, A. Madaan, S. Zhou, U. Alon, P. Liu, Y. Yang, J. Callan, and G. Neubig. Pal: Program-aided
language models. In International Conference on Machine Learning, pages 10764–10799. PMLR,
2023. URL https://arxiv.org/pdf/2211.10435.
Z. Gou, Z. Shao, Y. Gong, Y. Yang, M. Huang, N. Duan, W. Chen, et al. Tora: A tool-integrated
reasoning agent for mathematical problem solving. ArXiv preprint, abs/2309.17452, 2023. URL
https://arxiv.org/abs/2309.17452.
D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Measur-
ing massive multitask language understanding. In 9th International Conference on Learning
Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021, 2021a.
URL https:
//openreview.net/forum?id=d7KBjmI3GmQ.
D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt.
Measuring mathematical problem solving with the math dataset. In Thirty-fifth Conference on
Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021b. URL
https://openreview.net/forum?id=7Bywt2mQsCe.
H. Ivison, Y. Wang, V. Pyatkin, N. Lambert, M. Peters, P. Dasigi, J. Jang, D. Wadden, N. A. Smith,
I. Beltagy, et al. Camels in a changing climate: Enhancing lm adaptation with tulu 2. ArXiv
preprint, abs/2311.10702, 2023. URL https://arxiv.org/pdf/2311.10702.
A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. d. l. Casas, F. Bressand,
G. Lengyel, G. Lample, L. Saulnier, et al. Mistral 7b. ArXiv preprint, abs/2310.06825, 2023. URL
https://arxiv.org/abs/2310.06825.
A. Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bamford, D. S. Chaplot, D. d. l.
Casas, E. B. Hanna, F. Bressand, et al. Mixtral of experts. ArXiv preprint, abs/2401.04088, 2024.
URL https://arxiv.org/abs/2401.04088.
A. Joulin, E. Grave, P. Bojanowski, M. Douze, H. Jégou, and T. Mikolov. Fasttext. zip: Compressing
text classification models. ArXiv preprint, abs/1612.03651, 2016. URL https://arxiv.org/
abs/1612.03651.
A. Lewkowycz, A. Andreassen, D. Dohan, E. Dyer, H. Michalewski, V. Ramasesh, A. Slone,
C. Anil, I. Schlag, T. Gutman-Solo, et al. Solving quantitative reasoning problems with language
models. Advances in Neural Information Processing Systems, 35:3843–3857, 2022. URL https:
//openreview.net/forum?id=IFXTZERXdM7.
C. Li, W. Wang, J. Hu, Y. Wei, N. Zheng, H. Hu, Z. Zhang, and H. Peng. Common 7b language
models already possess strong math capabilities. ArXiv preprint, abs/2403.04706, 2024a. URL
https://arxiv.org/abs/2403.04706.
T. Li, W.-L. Chiang, L. D. Evan Frick, B. Zhu, J. E. Gonzalez, and I. Stoica. From live data to
high-quality benchmarks: The arena-hard pipeline, April 2024b. URL https://lmsys.org/
blog/2024-04-19-arena-hard/.
X. Li, T. Zhang, Y. Dubois, R. Taori, I. Gulrajani, C. Guestrin, P. Liang, and T. B. Hashimoto.
Alpacaeval: An automatic evaluator of instruction-following models. https://github.com/
tatsu-lab/alpaca_eval, 2023.
Z. Lin, Z. Gou, T. Liang, R. Luo, H. Liu, and Y. Yang. Criticbench: Benchmarking llms for critique-
correct reasoning. ArXiv preprint, abs/2402.14809, 2024. URL https://arxiv.org/abs/2402.
14809.
J. Liu, C. S. Xia, Y. Wang, and L. Zhang.
Is your code generated by chatgpt really correct?
rigorous evaluation of large language models for code generation. Advances in Neural Information
Processing Systems, 36, 2024. URL https://openreview.net/pdf?id=1qvx610Cu7.
H. Luo, Q. Sun, C. Xu, P. Zhao, J. Lou, C. Tao, X. Geng, Q. Lin, S. Chen, and D. Zhang. Wizardmath:
Empowering mathematical reasoning for large language models via reinforced evol-instruct. ArXiv
preprint, abs/2308.09583, 2023. URL https://arxiv.org/abs/2308.09583.
11


Meta. Introducing meta llama 3: The most capable openly available llm to date. https://ai.meta.
com/blog/meta-llama-3/, April 2024.
A. Mitra, H. Khanpour, C. Rosset, and A. Awadallah. Orca-math: Unlocking the potential of slms
in grade school math. ArXiv preprint, abs/2402.14830, 2024. URL https://arxiv.org/abs/
2402.14830.
M. Nye, A. J. Andreassen, G. Gur-Ari, H. Michalewski, J. Austin, D. Bieber, D. Dohan,
A. Lewkowycz, M. Bosma, D. Luan, et al.
Show your work: Scratchpads for intermedi-
ate computation with language models. In Deep Learning for Code Workshop, 2022. URL
https://openreview.net/forum?id=iedYJm92o0a.
L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agar-
wal,
K. Slama,
A. Ray,
et al.
Training language models to follow instructions
with human feedback.
Advances in neural information processing systems, 35:27730–
27744, 2022. URL https://proceedings.neurips.cc/paper_files/paper/2022/file/
b1efde53be364a73914f58805a001731-Paper-Conference.pdf.
K. Paster, M. Dos Santos, Z. Azerbayev, and J. Ba. Openwebmath: An open dataset of high-quality
mathematical web text. In The Twelfth International Conference on Learning Representations,
2023. URL https://openreview.net/pdf?id=jKHmjlpViu.
B. Peng, C. Li, P. He, M. Galley, and J. Gao. Instruction tuning with gpt-4. ArXiv preprint,
abs/2304.03277, 2023. URL https://arxiv.org/abs/2304.03277.
J. Rasley, S. Rajbhandari, O. Ruwase, and Y. He. Deepspeed: System optimizations enable training
deep learning models with over 100 billion parameters. In R. Gupta, Y. Liu, J. Tang, and B. A.
Prakash, editors, KDD ’20: The 26th ACM SIGKDD Conference on Knowledge Discovery and
Data Mining, Virtual Event, CA, USA, August 23-27, 2020, pages 3505–3506, 2020.
URL
https://dl.acm.org/doi/10.1145/3394486.3406703.
D. Rein, B. L. Hou, A. C. Stickland, J. Petty, R. Y. Pang, J. Dirani, J. Michael, and S. R. Bowman.
Gpqa: A graduate-level google-proof q&a benchmark. ArXiv preprint, abs/2311.12022, 2023.
URL https://arxiv.org/abs/2311.12022.
V. Sanh, A. Webson, C. Raffel, S. H. Bach, L. Sutawika, Z. Alyafeai, A. Chaffin, A. Stiegler, A. Raja,
M. Dey, M. S. Bari, C. Xu, U. Thakker, S. S. Sharma, E. Szczechla, T. Kim, G. Chhablani, N. V.
Nayak, D. Datta, J. Chang, M. T. Jiang, H. Wang, M. Manica, S. Shen, Z. X. Yong, H. Pandey,
R. Bawden, T. Wang, T. Neeraj, J. Rozen, A. Sharma, A. Santilli, T. Févry, J. A. Fries, R. Teehan,
T. L. Scao, S. Biderman, L. Gao, T. Wolf, and A. M. Rush. Multitask prompted training enables
zero-shot task generalization. In The Tenth International Conference on Learning Representations,
ICLR 2022, Virtual Event, April 25-29, 2022, 2022. URL https://openreview.net/forum?
id=9Vrb9D0WI4.
Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, M. Zhang, Y. Li, Y. Wu, and D. Guo. Deepseekmath: Push-
ing the limits of mathematical reasoning in open language models. ArXiv preprint, abs/2402.03300,
2024. URL https://arxiv.org/abs/2402.03300.
A. Srivastava, A. Rastogi, A. Rao, A. A. M. Shoeb, A. Abid, A. Fisch, A. R. Brown, A. Santoro,
A. Gupta, A. Garriga-Alonso, et al. Beyond the imitation game: Quantifying and extrapolating
the capabilities of language models. Transactions on Machine Learning Research, 2023. URL
https://openreview.net/forum?id=uyTL5Bvosj.
L. Sun, Y. Han, Z. Zhao, D. Ma, Z. Shen, B. Chen, L. Chen, and K. Yu. Scieval: A multi-level large
language model evaluation benchmark for scientific research. ArXiv preprint, abs/2308.13149,
2023. URL https://arxiv.org/abs/2308.13149.
M. Suzgun, N. Scales, N. Schärli, S. Gehrmann, Y. Tay, H. W. Chung, A. Chowdhery, Q. V. Le, E. H.
Chi, D. Zhou, , and J. Wei. Challenging big-bench tasks and whether chain-of-thought can solve
them. ArXiv preprint, abs/2210.09261, 2022. URL https://arxiv.org/abs/2210.09261.
Z. Tang, X. Zhang, B. Wan, and F. Wei. Mathscale: Scaling instruction tuning for mathematical
reasoning. ArXiv preprint, abs/2403.02884, 2024. URL https://arxiv.org/abs/2403.02884.
12


R. Taori, I. Gulrajani, T. Zhang, Y. Dubois, X. Li, C. Guestrin, P. Liang, and T. B. Hashimoto.
Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/
stanford_alpaca, 2023.
R. Taylor, M. Kardas, G. Cucurull, T. Scialom, A. Hartshorn, E. Saravia, A. Poulton, V. Kerkez, and
R. Stojnic. Galactica: A large language model for science. ArXiv preprint, abs/2211.09085, 2022.
URL https://arxiv.org/abs/2211.09085.
G. Team, R. Anil, S. Borgeaud, Y. Wu, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M.
Dai, A. Hauth, et al. Gemini: a family of highly capable multimodal models. ArXiv preprint,
abs/2312.11805, 2023. URL https://arxiv.org/abs/2312.11805.
Teknium. Openhermes 2.5: An open dataset of synthetic data for generalist llm assistants, 2023.
URL https://huggingface.co/datasets/teknium/OpenHermes-2.5.
TIGER-Lab. Mmlu professional dataset. Hugging Face Dataset Hub, 2024. URL https://
huggingface.co/datasets/TIGER-Lab/MMLU-Pro.
S. Toshniwal, I. Moshkov, S. Narenthiran, D. Gitman, F. Jia, and I. Gitman. Openmathinstruct-1:
A 1.8 million math instruction tuning dataset. ArXiv preprint, abs/2402.10176, 2024. URL
https://arxiv.org/abs/2402.10176.
H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal,
E. Hambro, F. Azhar, et al. Llama: Open and efficient foundation language models. ArXiv preprint,
abs/2302.13971, 2023a. URL https://arxiv.org/abs/2302.13971.
H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra,
P. Bhargava, S. Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. ArXiv
preprint, abs/2307.09288, 2023b. URL https://arxiv.org/abs/2307.09288.
X. Wang, Z. Hu, P. Lu, Y. Zhu, J. Zhang, S. Subramaniam, A. Loomba, S. Zhang, Y. Sun, and
W. Wang. Scibench: Evaluating college-level scientific problem-solving abilities of large language
models. In The 3rd Workshop on Mathematical Reasoning and AI at NeurIPS’23, 2023a. URL
https://openreview.net/forum?id=u6jbcaCHqO.
Y. Wang, S. Mishra, P. Alipoormolabashi, Y. Kordi, A. Mirzaei, A. Naik, A. Ashok, A. S.
Dhanasekaran, A. Arunkumar, D. Stap, E. Pathak, G. Karamanolakis, H. Lai, I. Purohit, I. Mon-
dal, J. Anderson, K. Kuznia, K. Doshi, K. K. Pal, M. Patel, M. Moradshahi, M. Parmar,
M. Purohit, N. Varshney, P. R. Kaza, P. Verma, R. S. Puri, R. Karia, S. Doshi, S. K. Sampat,
S. Mishra, S. Reddy A, S. Patro, T. Dixit, and X. Shen. Super-NaturalInstructions: General-
ization via declarative instructions on 1600+ NLP tasks. In Proceedings of the 2022 Confer-
ence on Empirical Methods in Natural Language Processing, pages 5085–5109, 2022. URL
https://aclanthology.org/2022.emnlp-main.340.
Y. Wang, Y. Kordi, S. Mishra, A. Liu, N. A. Smith, D. Khashabi, and H. Hajishirzi. Self-instruct:
Aligning language models with self-generated instructions. In Proceedings of the 61st Annual
Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13484–
13508, 2023b. URL https://arxiv.org/abs/2212.10560.
Z. Wang, R. Xia, and P. Liu. Generative ai for math: Part i–mathpile: A billion-token-scale pretraining
corpus for math. ArXiv preprint, abs/2312.17120, 2023c. URL https://arxiv.org/abs/2312.
17120.
J. Wei, M. Bosma, V. Y. Zhao, K. Guu, A. W. Yu, B. Lester, N. Du, A. M. Dai, and Q. V. Le.
Finetuned language models are zero-shot learners. In The Tenth International Conference on
Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022, 2022a. URL https:
//openreview.net/forum?id=gEZrGCozdqR.
J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhou, et al. Chain-of-thought
prompting elicits reasoning in large language models. Advances in neural information processing
systems, 35:24824–24837, 2022b. URL https://openreview.net/pdf?id=_VjQlMeSB_J.
13


C. Xu, Q. Sun, K. Zheng, X. Geng, P. Zhao, J. Feng, C. Tao, and D. Jiang. Wizardlm: Empowering
large language models to follow complex instructions. ArXiv preprint, abs/2304.12244, 2023. URL
https://arxiv.org/abs/2304.12244.
H. Ying, S. Zhang, L. Li, Z. Zhou, Y. Shao, Z. Fei, Y. Ma, J. Hong, K. Liu, Z. Wang, et al.
Internlm-math: Open math large language models toward verifiable reasoning. ArXiv preprint,
abs/2402.06332, 2024. URL https://arxiv.org/abs/2402.06332.
A. Young, B. Chen, C. Li, C. Huang, G. Zhang, G. Zhang, H. Li, J. Zhu, J. Chen, J. Chang,
et al. Yi: Open foundation models by 01. ai. ArXiv preprint, abs/2403.04652, 2024. URL
https://arxiv.org/abs/2403.04652.
L. Yu, W. Jiang, H. Shi, Y. Jincheng, Z. Liu, Y. Zhang, J. Kwok, Z. Li, A. Weller, and W. Liu.
Metamath: Bootstrap your own mathematical questions for large language models. In The Twelfth
International Conference on Learning Representations, 2023. URL https://openreview.net/
forum?id=N8N0hgNDRt.
L. Yuan, G. Cui, H. Wang, N. Ding, X. Wang, J. Deng, B. Shan, H. Chen, R. Xie, Y. Lin, et al.
Advancing llm reasoning generalists with preference trees. ArXiv preprint, abs/2404.02078, 2024.
URL https://arxiv.org/abs/2404.02078.
X. Yue, Y. Ni, K. Zhang, T. Zheng, R. Liu, G. Zhang, S. Stevens, D. Jiang, W. Ren, Y. Sun, et al.
Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert
agi. ArXiv preprint, abs/2311.16502, 2023a. URL https://arxiv.org/abs/2311.16502.
X. Yue, X. Qu, G. Zhang, Y. Fu, W. Huang, H. Sun, Y. Su, and W. Chen. Mammoth: Building math
generalist models through hybrid instruction tuning. In The Twelfth International Conference on
Learning Representations, 2023b. URL https://openreview.net/forum?id=yLClGs770I.
Y. Zhang, Y. Luo, Y. Yuan, and A. C.-C. Yao. Automathtext: Autonomous data selection with
language models for mathematical texts. ArXiv preprint, abs/2402.07625, 2024. URL https:
//arxiv.org/abs/2402.07625.
L. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. Xing,
et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information
Processing Systems, 36, 2024a. URL https://openreview.net/forum?id=uccHPGDlao.
T. Zheng, S. Guo, X. Qu, J. Guo, W. Zhang, X. Du, C. Lin, W. Huang, W. Chen, J. Fu, et al. Kun:
Answer polishment for chinese self-alignment with instruction back-translation. ArXiv preprint,
abs/2401.06477, 2024b. URL https://arxiv.org/abs/2401.06477.
T. Zheng, G. Zhang, T. Shen, X. Liu, B. Y. Lin, J. Fu, W. Chen, and X. Yue. Opencodeinterpreter:
Integrating code generation with execution and refinement. ArXiv preprint, abs/2402.14658, 2024c.
URL https://arxiv.org/abs/2402.14658.
Y. Zheng, R. Zhang, J. Zhang, Y. Ye, Z. Luo, and Y. Ma. Llamafactory: Unified efficient fine-tuning
of 100+ language models. ArXiv preprint, abs/2403.13372, 2024d. URL https://arxiv.org/
abs/2403.13372.
14


A
Details of Additional Public Instruction Tuning Datasets
• OpenHermes 2.5 (Teknium, 2023): The OpenHermes-2.5 dataset is a comprehensive collection
of diverse data sources for instruction tuning, including 1M examples from datasets in math,
science, and coding, alongside synthetic and chat-based data. It incorporates diverse sources such
as Airoboros 2.2, CamelAI Domain Expert Datasets, ChatBot Arena, Collective Cognition, Evol
Instruct, Glaive Code Assistant, GPT4-LLM, GPTeacher, Medical Tasks, MetaMath, SlimOrca,
Platypus, ShareGPT, and Unnatural Instructions GPT4. We remove TheoremQA from Platypus as
it is one of our test sets.
• Code-Feedback (Zheng et al., 2024c): The Code-Feedback dataset is a multi-turn code generation
and refinement dataset, containing 68,000 multi-turn interactions between users, code generation
models, and compiler systems. It includes initial user instructions followed by compiler and user
feedback. This dataset significantly enhances the model’s multi-turn interaction coding ability.
• Math-Plus: This dataset combines public datasets such as MetaMathQA (395K) (Yu et al., 2023)
and Orca-Math (200K) (Mitra et al., 2024). Both of these datasets are generated by GPT-3.5/4
using GSM/MATH and other math datasets as the seed data. To further augment the dataset, we
prompt GPT-4 to rewrite Q-A pairs from MATH training sets, adding an additional 300K examples
to enhance the challenging problems. The total size of the Math-Plus dataset is 894K examples.
To ensure consistency and compatibility, we carefully align the format and structure of these additional
datasets with WEBINSTRUCT. These supplementary datasets provide a rich resource for training
models to answer questions and provide explanations across a wide range of topics, enhancing their
versatility and applicability in real-world scenarios.
B
Details of Evaluation Datasets
To rigorously assess the capabilities of models in reasoning abilities across different domains, we
utilize several widely used datasets. Each of these datasets is designed to challenge the models in
different aspects of reasoning.
• GSM8K (Cobbe et al., 2021): This test dataset contains 1.32K diverse grade school math problems,
intended to test basic arithmetic and reasoning ability in an educational context.
• MATH (Hendrycks et al., 2021b): Comprising 5000 intricate competition-level problems to
evaluate the models’ ability to perform complex mathematical reasoning.
• TheoremQA (Chen et al., 2023b): Focused on applying mathematical theorems to solve advanced
problems in fields such as mathematics, physics, and engineering, TheoremQA includes 800
questions that test the theoretical reasoning capabilities.
• BIG-Bench Hard (BBH) (Suzgun et al., 2022): Consisting of 23 tasks previously found challeng-
ing for language models from BIG-Bench (Srivastava et al., 2023), BBH contains a total of 6511
challenging problems examining the capability of LLMs to solve them.
• ARC-C (Clark et al., 2018): ARC includes questions derived from various grade-level science
exams, testing models’ ability to handle both straightforward and complex scientific queries. We
use the challenge subset, which contains 1,172 test questions.
• GPQA (Rein et al., 2023): This dataset provides "Google-proof" questions in biology, physics, and
chemistry, designed to test deep domain expertise and reasoning under challenging conditions. We
use the diamond subset containing 198 hard problems.
• MMLU-STEM (Hendrycks et al., 2021a): Spanning 57 subjects across multiple disciplines,
MMLU evaluates the breadth and depth of a model’s knowledge in a manner akin to academic and
professional testing environments. We select the STEM subset of MMLU with 3.13K problems.
These datasets collectively enable a comprehensive assessment of language models’ reasoning
prowess across a spectrum of complexity and realism. We further evaluate the models on additional
code generation tasks (including HumanEval (Chen et al., 2021), MBPP (Austin et al., 2021) and
their augmented version (Liu et al., 2024)), general LLM benchmarks like MMLU (Hendrycks
et al., 2021a), and chat benchmarks like MT-Bench (Zheng et al., 2024a), AlpacaEval 2.0 (Li et al.,
2023), and Arena Hard (Li et al., 2024b) to demonstrate the generalizability of WEBINSTRUCT and
WEBINSTRUCT-PLUS on more general LLM benchmarks.
15


C
Related Work
Instruction Tuning.
Instruction tuning is crucial for aligning large language models (LLMs) with
end tasks. There are two main types of instruction tuning data: (1) human-written data, such as
FLAN (Wei et al., 2022a), T0 (Sanh et al., 2022), and SuperNI (Wang et al., 2022), which assemble
large instruction-tuning datasets from existing human-labeled datasets; and (2) synthesized data, like
Self-Instruct (Wang et al., 2023b), WizardLM (Xu et al., 2023), and GPT4-Alpaca (Peng et al., 2023),
which create instruction-tuning datasets by synthesizing from powerful LLMs like GPT-4 (Achiam
et al., 2023). Both types of instruction-tuning data have advantages and disadvantages. Human-
written data is limited in size due to the high cost and in task diversity because existing human-labeled
datasets mostly focus on a few NLP tasks. Although synthesized data can be generated at any scale,
the high rate of hallucination can lead to significant quality degradation. Moreover, the diversity of
synthesized data is heavily influenced by the seed data. Without a diverse seed dataset, the synthesized
data will lack domain coverage.
Mathematics Reasoning.
In recent years, there has been a growing interest in enhancing the
mathematical reasoning abilities of large language models (LLMs). Three main approaches have
been proposed to improve LLMs’ mathematical reasoning skills:
• Prompting: Chain-of-thought-prompting (CoT) (Nye et al., 2022; Wei et al., 2022b) elicits LLMs’
inherent reasoning ability by demonstrating intermediate reasoning steps. Program-of-thoughts-
prompting (PoT) (Chen et al., 2023a; Gao et al., 2023) utilizes tools to further augment LLMs’
math reasoning abilities. Subsequent work (Gou et al., 2023; Toshniwal et al., 2024; Yuan et al.,
2024) combines CoT and PoT to maximize LLMs’ reasoning ability.
• Continued Training: Enabling LLMs to solve mathematical problems has been a long-standing
challenge. MINERVA (Lewkowycz et al., 2022) and Galactica (Taylor et al., 2022) were pioneers
in continued training of LLMs to adapt to scientific domains for math and science reasoning.
Open-source models like Llemma (Azerbayev et al., 2023), DeepSeek-Math (Shao et al., 2024),
and Intern-Math (Ying et al., 2024) have surpassed MINERVA and Galactica on math benchmarks.
These approaches mainly rely on using an efficient classifier to recall documents from Common
Crawl to retrieve a massive high-quality math-related corpus (>100B tokens) to enhance math
reasoning.
• Instruction Tuning: Instruction tuning aims to enhance LLMs’ math reasoning skills by efficiently
training on human-annotated public datasets like GSM8K (Cobbe et al., 2021), MATH (Hendrycks
et al., 2021b), and MathQA (Amini et al., 2019). However, these datasets are often insufficient
in size and diversity. Therefore, recent work (Yu et al., 2023; Yue et al., 2023b; Toshniwal et al.,
2024; Luo et al., 2023; Li et al., 2024a) proposes augmenting them with strong commercial LLMs
like GPT-4 (Achiam et al., 2023). These methods can significantly boost LLMs’ performance on
in-domain math benchmarks but may fall short of generalization.
Our work combines continued training with instruction tuning to exploit the benefits of both ap-
proaches. Specifically, our dataset is recalled from Common Crawl like DeepSeekMath. However,
due to the significant level of noise in the raw corpus, we utilize a strong LLM to filter and clean the
corpus to extract the instruction tuning pairs.
Science Reasoning.
In addition to mathematical reasoning, there is growing interest in improving
LLMs’ general scientific reasoning ability in subjects like physics, biology, chemistry, computer
science, etc. Several benchmarks, such as MMLU (Hendrycks et al., 2021a), TheoremQA (Chen et al.,
2023b), Sci-Bench (Wang et al., 2023a), SciEval (Sun et al., 2023), and GPQA (Rein et al., 2023),
have been developed to measure LLMs’ reasoning ability on tasks beyond math. However, there
has been less effort in curating high-quality training data for the science domain. Most datasets, like
OpenWebMath (Paster et al., 2023), Proof-Pile (Azerbayev et al., 2023), and MathPile (Zhang et al.,
2024), are heavily biased towards mathematics. In this work, we aim to generalize the pre-training
data to broader subjects through our newly curated science seed data.
16


D
Code Generation Results
We report the code generation results of our models and baselines in Table 6.
HumanEval
HumanEval+
MBPP
MBPP+
Average
Average+
Mistral-7B
28.7
23.8
51.9
42.1
40.3
33.0
Gemma-7B
26.8
20.1
52.6
43.4
39.7
31.8
Llama-3-8B
33.5
29.3
61.4
51.6
47.5
40.5
Gemma-1.1-7B-Instruct
42.7
35.4
57.1
45.0
49.9
40.2
Mistral-7B-Instruct-v0.2
75.0
70.1
44.7
37.0
59.9
53.6
Llama-3-8B-Instruct
61.6
56.7
70.1
59.3
65.9
58.0
Mixtral-8×7B-Instruct-v0.1
45.1
39.6
59.5
49.7
52.3
44.7
MAmmoTH2– 7B-Plus
72.1
65.9
60.1
50.4
66.1
58.2
MAmmoTH2– 8B-Plus
63.4
57.9
60.4
48.6
61.9
53.3
MAmmoTH2– 8×7B-Plus
57.9
53.7
68.7
56.9
63.3
55.3
Table 6: Code generation results of different models.
Baseline results are copied from the
EvalPlus (Liu et al., 2024) leaderboard.
E
Impact of Additional Public Instruction Tuning Datasets
In Appendix A, we introduce additional public instruction tuning datasets to further boost the model’s
reasoning performance. Here, we show the three setups of models trained on: 1) WEBINSTRUCT
only; 2) Additional Public Datasets only; 3) WEBINSTRUCT + Additional Public Datasets (which
we first trained on WEBINSTRUCT and then continued training on additional public datasets). The
results are shown in Table 7.
Data
TheoremQA
MATH
GSM8K
GPQA
MMLU-ST
BBH
ARC-C
AVG
Mistral 7B Base
WEBINSTRUCT
29.0
36.7
68.4
32.4
62.4
58.6
81.7
52.8
PUBLIC DATASETS
22.6
37.9
83.5
29.3
57.6
62.7
79.9
53.4
WEBINS.+PUBLIC.
29.2
45.0
84.7
36.8
64.5
63.1
83.0
58.0
Mixtral 8x7B Base
WEBINSTRUCT
32.2
39.0
75.4
36.8
67.4
71.1
87.5
58.9
PUBLIC DATASETS
31.3
45.1
85.3
37.4
69.4
73.2
88.1
61.4
WEBINS.+PUBLIC.
34.1
47.0
86.4
37.8
72.4
74.1
88.4
62.9
Table 7: Impact of additional public instruction tuning datasets.
F
Distributions of Website Domains in WEBINSTRUCT
Figure 7 show the distribution of the top websites in WEBINSTRUCT.
17


Figure 7: The distribution of the top websites in our instruction dataset.
G
Domain Distribution of WEBINSTRUCT
Figure 8 presents a breakdown of the WEBINSTRUCT by subject domains and data sources, providing
insights into the composition and diversity of the mined instruction-response pairs. The subject labels
are automatically annotated using the Llama-3-8B-Instruct model, while the distribution between
education and forum data is obtained by analyzing the source URLs of the samples. The pie chart
reveals that WEBINSTRUCT is predominantly composed of science-related subjects, with 81.69%
of the data falling under the broad "Science" category. Within this category, Mathematics takes
up the largest share at 68.36%, followed by Physics, Chemistry, and Biology. This highlights the
dataset’s strong emphasis on mathematical problem-solving and scientific reasoning. The remaining
non-science categories, such as Business, Art & Design, and Health & Medicine, contribute to the
diversity of the dataset. In terms of data sources, the vast majority (86.73%) of the instruction-
response pairs come from exam-style questions, while forum discussions make up the remaining
13.27%. This source breakdown indicates that WEBINSTRUCT primarily consists of well-structured,
educational content, supplemented by real-world discussions and inquiries from forums. The diverse
subject coverage and the combination of education and forum data enable WEBINSTRUCT to capture
a wide range of reasoning tasks and problem-solving scenarios.
Figure 8: Breakdown of WEBINSTRUCT by subject domains and data sources.
18


H
Limitations of WEBINSTRUCT
Despite employing a three-step pipeline to ensure the quality of the mined instruction-response pairs,
there may still be some noise and inaccuracies in the dataset, as mentioned in Figure 6. The extraction
and refinement steps rely on the performance of the LLMs used, which may introduce biases and
errors. Future work could explore more advanced techniques for data cleaning and validation, such
as human-in-the-loop approaches or training a data selection model for filtering. Furthermore, while
WEBINSTRUCT covers a wide range of subjects, including math, science, and engineering, there
may be specific subdomains or advanced topics that are underrepresented, such as humanities and
other daily chat topics. Expanding the coverage of the seed data to include broader and more diverse
instruction-response pairs could further enhance the reasoning capabilities of LLMs in different areas.
I
Broader Social Impact
The development of MAmmoTH2 and the WEBINSTRUCT has the potential to positively impact edu-
cation by providing the community with access to a large-scale, diverse set of instruction-response
pairs across various subjects, particularly in mathematics and science. MAmmoTH2 can assist students
in their learning process by offering step-by-step solutions, explanations, and targeted feedback,
thus enhancing their understanding of complex concepts and problem-solving skills. Furthermore,
the dataset and models can serve as valuable resources for educators to create more effective and
engaging educational content, assignments, and assessments. However, it is important to consider
potential limitations and biases in the dataset and the model. The quality and accuracy of the
instruction-response pairs in WEBINSTRUCT may vary, and the model’s outputs may reflect these
inconsistencies. Therefore, it is crucial to employ rigorous data filtering and quality control measures
to ensure the reliability and trustworthiness of the dataset and the model.
19


J
Case Study
In this section, we aim to review some of the instances in WEBINSTRUCT. The data extracted from
the website contains many issues. For example, the most common issue is that the answer is relatively
short without any intermediate rationale, which could occur frequently from the quiz website with
multi-choice questions. Another common issue is that the extraction destroys important formatting in
the document, which leads to unrecoverable errors.
Good Case 1
Extracted:
Question: Find the distance from the point 3
2, 3
2 to the line −37x
10 −21y
10 −7
2 = 0.
Answer: 61
q
2
905.
Refined:
Question:
Calculate the distance from the point ( 3
2, 3
2) to the line defined by the equation
−37
10x −21
10y −7
2 = 0.
Answer: To find the distance ’d’ from a point (x0, y0) to a line given by Ax + By + C = 0, we use
the formula:
d = |Ax0 + By0 + C|
√
A2 + B2
.
For the point
 3
2, 3
2


and the line −37
10x −21
10y −7
2 = 0, substitute the coordinates into the formula:
d = | −37
10 · 3
2 −21
10 · 3
2 −7
2|
q−37
10

2 +
−21
10

2 .
Simplifying further, the exact simplified distance ’d’ is 61
q
2
905. Thus, the correct distance from the
point to the line is 61
q
2
905.
Figure A1: In this Refined version, the expression of both the question and the answer is clearer and
more precise. It employs standardized mathematical symbols and terms, such as representing the
coordinates of the point and the equation of the line as (x0, y0) and Ax + By + C = 0, respectively.
Moreover, it explicitly provides the formula for calculating the distance. Through these revisions,
readers can more easily understand how to compute the distance from a point to a line, making the
solution process more transparent and easier to follow.
Good Case 2
Extracted:
Question: What are the two main ways that minerals are classified?
Answer: They can be classified by luster, streak, breakage, and hardness.
Refined:
Question: What are the primary criteria used for classifying minerals in mineralogy?
Answer: Minerals are primarily classified based on two main categories: physical properties and
chemical properties. The physical properties include luster, streak, crystal form, hardness, density, and
cleavage or breakage. Chemical properties involve the mineral’s chemical composition and reactions with
other substances. These characteristics help identify and distinguish different minerals in mineralogy.
Figure A2: Refining the extracted response provides a more comprehensive and accurate explanation
of mineral classification in mineralogy. By expanding on the initial response to include both physical
and chemical properties, the refined version offers a deeper understanding of how minerals are
categorized and identified. This enriched content can enhance the model’s ability to generate
informative responses related to mineralogy.
20


Good Case 3
Extracted:
Question: How do you find theasymptotes for Q(x) = (2x2)/(x2 −5x −6)?
Answer: vertical asymptotes x=-1, x=6/n horizontal asymptote y=2 /n /n /n /n /n /n /n /n /n /n
Refined:
Question: How do you determine the vertical and horizontal asymptotes for the function Q(x) =
2x2
x2−5x−6?
Answer: To find the vertical asymptotes, we need to identify the values of x that make the denominator
equal to zero:
x2 −5x −6 = (x −6)(x + 1) = 0
This gives us the vertical asymptotes at x = −1 and x = 6.
For the horizontal asymptote, as x approaches infinity or negative infinity, the dominant terms in the
numerator and denominator are both x2. Thus, we compare their coefficients:
- Numerator’s highest degree coefficient: 2 - Denominator’s highest degree coefficient: 1
The horizontal asymptote is given by the ratio of these coefficients:
y = 2
1 = 2
Therefore, the vertical asymptotes are at x = −1 and x = 6, and the horizontal asymptote is y = 2.
Figure A3: The refined answer provides a more detailed explanation for finding the asymptotes. It
includes the steps for solving for the vertical asymptotes by factoring the denominator and determining
where it equals zero. It also explains how to find the horizontal asymptote by comparing the leading
coefficients of the highest-degree terms in both the numerator and the denominator. The use of LaTeX
formatting makes the mathematical notation clearer and easier to understand.
Good Case 4
Extracted:
Question:Which of the following is not an example of biased language? a. “You are too short to reach
that.” b. “Are you deaf?” c. “Move along, grandpa!” d. “I think I’ve been gypped.
Answer: D. I think I’ve been gypped.
Refined:
Question: Which of the following statements does not exhibit biased language?
a. “You are too short to reach that.”
b. “Are you deaf?”
c. “Move along, grandpa!”
d. “I think I’ve been gypped.”
Answer: The statement that does not exhibit biased language is:
d. "I think I’ve been gypped."
Explanation: Biased language refers to words or phrases that stereotype, discriminate, or show prejudice
towards a particular group. In this case, options a, b, and c contain biases towards people’s physical
abilities (height), hearing ability, and age, respectively. However, "I think I’ve been gypped" is an
informal expression meaning to be cheated or deceived, which, while potentially offensive, does not
directly target a specific group based on attributes like race, gender, or disabilities.
Figure A4: The refined version of the input presents the question and answers in a clearer, more
structured format that improves readability and comprehension. Specifically, using a bulleted list
for the options makes them distinct and easier to evaluate individually. Moreover, the addition of
an explanation provides context and justification for the selected answer, enhancing the educational
value of the content. This refined structure and the inclusion of an explanatory note make it a superior
training example for models, as it helps in understanding not just the answer but the rationale behind
it, promoting deeper learning and better pattern recognition in language processing tasks.
21


Good Case 5
Extracted:
Question: I am stucked on the following challenge: "If the line determined by two distinct points
(x1, y1) and (x2, y2) is not vertical, and therefore has slope (y2 −y1)/(x2 −x1), show that the point-
slope form of its equation is the same regardless of which point is used as the given point." Okay, we
can separate (x0, y0) from the form to get:
y(x2 −x1) −x(y2 −y1) = y0(x2 −x1) −x0(y2 −y1)
But how exclude this point (x0, y0) and leave only x, y, x1, y1, x2, y2 in the equation? UPDATE: There
is a solution for this challenge:
(y1 −y2)x + (x2 −x1)y = x2y1 −x1y2
From the answer I found that
y2(x −x1) −y1(x −x2) = y(x2 −x1)
... but why this is true?
Answer: Thanks to saulspatz, the solution is to simply show that, whether we are using (x1, y1) or
(x2, y2) as the given point, the equation does not change.
So both equations:
y −y1 = m(x −x1)
y −y2 = m(x −x2)
reduce to:
(y1 −y2)x + (x2 −x1)y = x2y1 −x1y2.
Refined:
Question: Demonstrate that the point-slope form of the equation of a line determined by two distinct
points (x1, y1) and (x2, y2), where the line is not vertical, remains consistent regardless of which point
is chosen as the reference. Why does the form of the equation not change when different reference
points are used?
Answer: To address this challenge, we need to confirm that the point-slope equation of a line remains
consistent regardless of the point chosen to derive it. Given two points (x1, y1) and (x2, y2) with a
non-vertical line passing through them, the slope m of the line is y2−y1
x2−x1 .
The point-slope form of a line’s equation based on point (x1, y1) is:
y −y1 = m(x −x1)
Similarly, using point (x2, y2) gives:
y −y2 = m(x −x2)
These can be rewritten and compared to show their equivalence. Expanding both equations: - From the
first equation: y −y1 = y2−y1
x2−x1 (x −x1) - From the second equation: y −y2 = y2−y1
x2−x1 (x −x2)
We simplify these to find a common form. Multiplying both sides by (x2 −x1) and rearranging terms,
we find:
y2(x −x1) −y1(x −x2) = y(x2 −x1)
This confirms that both forms collapse to the same linear equation, showing the consistency of the
point-slope form irrespective of the chosen reference point. Thus, it is demonstrated that the structural
form of the equation remains unchanged, ensuring that the mathematical properties of the line—defined
by its slope and points through which it passes—are invariant to the choice of the reference point used
in its expression. This uniformity stems from the linearity and the constant slope defined by any two
points on a straight line.
Figure A5: The original answer contains an irrelevant introductory sentence, which makes it less
focused. In contrast, the refined version provides a clearer question by explicitly stating the problem
of demonstrating the invariance of the point-slope form. The explanation in the answer is expanded
to show step-by-step equivalence between the two forms and includes detailed reasoning about why
the consistency holds. The added explanation reinforces the understanding of linear equations and
the concept of constant slope, making the solution more comprehensive and pedagogically valuable.
22


Bad Case 1
Extracted:
Question: Sort
n
−3π, 9, 17
√
3, −4, 47
7 , 3
e,
3
√
5
o
.
Answer:
n
−3π, −4, 3
e,
3
√
5, 47
7 , 9, 17
√
3
o
.
Refined:
Question: Sort the following numbers in ascending order: −3π, 9, 17
√
3, −4, 47
7 , 3
e,
3
√
5.
Answer: The sorted numbers in ascending order are: −3π, −4, 3
e,
3
√
5, 47
7 , 9, 17
√
3.
Figure A6: The refined example does not enhance the clarity or quality of the information. The
original formatting and question structure were already clear and concise, and the transformation
into a slightly more verbose question-and-answer format does not add significant value. Additionally,
both versions use proper LaTeX formatting for mathematical expressions, so there is no improvement
in readability or technical accuracy. Essentially, the refinement introduces additional words without
enhancing comprehension, which could potentially dilute the effectiveness of the model training by
focusing on unnecessary stylistic alterations rather than content clarity or quality.
Bad Case 2
Extracted:
Question: Which curve (AD or SRAS) will be affected and which direction (leftward or rightward)
will it shift when Congress goes to war against Canada?
Answer: When Congress falls into a war with Canada, the aggregate demand curve will be affected by
moving to the left. Suppose Congress fights against Canada, then, it is likely that Congress will pass
laws that advocate for increased tariffs on imports.
Spending on exports is an element of aggregate demand. Thus, its curve shifts rightwards, when its
components, which are, spending on consumptions, government spending, or spending on investment
increase.
On the other hand, Canada will not do business with the USA as usual when they fall out with Congress.
For example, the expenses on exports to Canada will reduce.
Thus, the aggregate demand will fail to increase and shift on the left if there is less consumption from
households, falls of investments from firms or when the demand for exports from a country like Canada
falls.
Refined:
Question: How does a war between Congress and Canada affect the Aggregate Demand (AD) curve,
and in which direction will it shift?
Answer: When Congress declares war against Canada, the
Aggregate Demand (AD) curve shifts to the left because Congress will implement increased tariffs on
imports. Spending on exports, which is a component of aggregate demand, decreases due to these tariffs.
The Short Run Aggregate Supply (SRAS) curve, however, is not directly affected by this situation. It
mainly reflects changes in production costs and the availability of resources, which are not directly
influenced by international conflicts unless they lead to disruptions in the domestic production process.
In summary, the AD curve shifts left due to decreased exports and potentially lower consumer and
business spending, while the SRAS curve remains relatively stable unless there are indirect impacts on
domestic production.
Figure A7: The refined answer, while more structured and clearer in terms of the specific economic
curves discussed, introduces inaccuracies that were not present in the original. Specifically, the
refined version inaccurately states that spending on exports decreases due to tariffs, which contradicts
the fundamental economic concept that tariffs affect imports rather than exports. Furthermore,
the original answer provides a broader analysis of potential economic impacts, such as changes
in household consumption and firm investment, which are omitted in the refined version. These
omissions reduce the depth and accuracy of the economic analysis provided in response to the effects
of war on economic curves.
23


Bad Case 3
Extracted:
Question: Is there a tool (LaTeX, JavaScript, Mathematica..) that allows one to draw Venn diagram
automatically from subsets relations, e.g. A ⊂A + B, A ⊂C, C ⊂C + D, B ̸⊂C would yield:
|
|
|
A+B
|
|
|
|
|
|
|
|
|
|
A
|
|
|
|
|
|
|
|
C+D
|

Abstract
Mathematical reasoning poses a significant challenge for language models due to its complex
and structured nature. In this paper, we introduce DeepSeekMath 7B, which continues pre-
training DeepSeek-Coder-Base-v1.5 7B with 120B math-related tokens sourced from Common
Crawl, together with natural language and code data. DeepSeekMath 7B has achieved an
impressive score of 51.7% on the competition-level MATH benchmark without relying on
external toolkits and voting techniques, approaching the performance level of Gemini-Ultra
and GPT-4. Self-consistency over 64 samples from DeepSeekMath 7B achieves 60.9% on MATH.
The mathematical reasoning capability of DeepSeekMath is attributed to two key factors: First,
we harness the significant potential of publicly available web data through a meticulously
engineered data selection pipeline. Second, we introduce Group Relative Policy Optimization
(GRPO), a variant of Proximal Policy Optimization (PPO), that enhances mathematical reasoning
abilities while concurrently optimizing the memory usage of PPO.
Figure 1 | Top1 accuracy of open-source models on the competition-level MATH benchmark
(Hendrycks et al., 2021) without the use of external toolkits and voting techniques.
∗Core contributors.
† Work done during internship at DeepSeek-AI.
arXiv:2402.03300v3  [cs.CL]  27 Apr 2024


1. Introduction
Large language models (LLM) have revolutionized the approach to mathematical reasoning
in artificial intelligence, spurring significant advancements in both the quantitative reasoning
benchmark (Hendrycks et al., 2021) and the geometry reasoning benchmark (Trinh et al., 2024).
Moreover, these models have proven instrumental in assisting humans in solving complex
mathematical problems (Tao, 2023). However, cutting-edge models such as GPT-4 (OpenAI,
2023) and Gemini-Ultra (Anil et al., 2023) are not publicly available, and the currently accessible
open-source models considerably trail behind in performance.
In this study, we introduce DeepSeekMath, a domain-specific language model that signifi-
cantly outperforms the mathematical capabilities of open-source models and approaches the
performance level of GPT-4 on academic benchmarks. To achieve this, we create the DeepSeek-
Math Corpus, a large-scale high-quality pre-training corpus comprising 120B math tokens. This
dataset is extracted from the Common Crawl (CC) using a fastText-based classifier (Joulin et al.,
2016). In the initial iteration, the classifier is trained using instances from OpenWebMath (Paster
et al., 2023) as positive examples, while incorporating a diverse selection of other web pages to
serve as negative examples. Subsequently, we employ the classifier to mine additional positive
instances from the CC, which are further refined through human annotation. The classifier is
then updated with this enhanced dataset to improve its performance. The evaluation results
indicate that the large-scale corpus is of high quality, as our base model DeepSeekMath-Base
7B achieves 64.2% on GSM8K (Cobbe et al., 2021) and 36.2% on the competition-level MATH
dataset (Hendrycks et al., 2021), outperforming Minerva 540B (Lewkowycz et al., 2022a). In
addition, the DeepSeekMath Corpus is multilingual, so we notice an improvement in Chinese
mathematical benchmarks (Wei et al., 2023; Zhong et al., 2023). We believe that our experience
in mathematical data processing is a starting point for the research community, and there is
significant room for improvement in the future.
DeepSeekMath-Base is initialized with DeepSeek-Coder-Base-v1.5 7B (Guo et al., 2024), as
we notice that starting from a code training model is a better choice compared to a general
LLM. Furthermore, we observe the math training also improves model capability on MMLU
(Hendrycks et al., 2020) and BBH benchmarks (Suzgun et al., 2022), indicating it does not only
enhance the model’s mathematical abilities but also amplifies general reasoning capabilities.
After pre-training, we apply mathematical instruction tuning to DeepSeekMath-Base with
chain-of-thought (Wei et al., 2022), program-of-thought (Chen et al., 2022; Gao et al., 2023), and
tool-integrated reasoning (Gou et al., 2023) data. The resulting model DeepSeekMath-Instruct
7B beats all 7B counterparts and is comparable with 70B open-source instruction-tuned models.
Furthermore, we introduce the Group Relative Policy Optimization (GRPO), a variant rein-
forcement learning (RL) algorithm of Proximal Policy Optimization (PPO) (Schulman et al., 2017).
GRPO foregoes the critic model, instead estimating the baseline from group scores, significantly
reducing training resources. By solely using a subset of English instruction tuning data, GRPO
obtains a substantial improvement over the strong DeepSeekMath-Instruct, including both
in-domain (GSM8K: 82.9% →88.2%, MATH: 46.8% →51.7%) and out-of-domain mathematical
tasks (e.g., CMATH: 84.6% →88.8%) during the reinforcement learning phase. We also provide
a unified paradigm to understand different methods, such as Rejection Sampling Fine-Tuning
(RFT) (Yuan et al., 2023a), Direct Preference Optimization (DPO) (Rafailov et al., 2023), PPO and
GRPO. Based on such a unified paradigm, we find that all these methods are conceptualized as
either direct or simplified RL techniques. We also conduct extensive experiments, e.g., online
v.s. offline training, outcome v.s. process supervision, single-turn v.s. iterative RL and so on,
2


to deeply investigate the essential elements of this paradigm. At last, we explain why our RL
boosts the performance of instruction-tuned models, and further summarize potential directions
to achieve more effective RL based on this unified paradigm.
1.1. Contributions
Our contribution includes scalable math pre-training, along with the exploration and analysis of
reinforcement learning.
Math Pre-Training at Scale
• Our research provides compelling evidence that the publicly accessible Common Crawl
data contains valuable information for mathematical purposes. By implementing a metic-
ulously designed data selection pipeline, we successfully construct the DeepSeekMath
Corpus, a high-quality dataset of 120B tokens from web pages filtered for mathemati-
cal content, which is almost 7 times the size of the math web pages used by Minerva
(Lewkowycz et al., 2022a) and 9 times the size of the recently released OpenWebMath
(Paster et al., 2023).
• Our pre-trained base model DeepSeekMath-Base 7B achieves comparable performance
with Minerva 540B (Lewkowycz et al., 2022a), indicating the number of parameters is not
the only key factor in mathematical reasoning capability. A smaller model pre-trained on
high-quality data could achieve strong performance as well.
• We share our findings from math training experiments. Code training prior to math
training improves models’ ability to solve mathematical problems both with and without
tool use. This offers a partial answer to the long-standing question: does code training
improve reasoning abilities? We believe it does, at least for mathematical reasoning.
• Although training on arXiv papers is common, especially in many math-related papers, it
brings no notable improvements on all mathematical benchmarks adopted in this paper.
Exploration and Analysis of Reinforcement Learning
• We introduce Group Relative Policy Optimization (GRPO), an efficient and effective
reinforcement learning algorithm. GRPO foregoes the critic model, instead estimating
the baseline from group scores, significantly reducing training resources compared to
Proximal Policy Optimization (PPO).
• We demonstrate that GRPO significantly enhances the performance of our instruction-
tuned model DeepSeekMath-Instruct, by solely using the instruction-tuning data. Further-
more, we observe enhancements in the out-of-domain performance during the reinforce-
ment learning process.
• We provide a unified paradigm to understand different methods, such as RFT, DPO,
PPO, and GRPO. We also conduct extensive experiments, e.g., online v.s. offline training,
outcome v.s. process supervision, single-turn v.s. iterative reinforcement learning, and so
on to deeply investigate the essential elements of this paradigm.
• Based on our unified paradigm, we explore the reasons behind the effectiveness of rein-
forcement learning, and summarize several potential directions to achieve more effective
reinforcement learning of LLMs.
1.2. Summary of Evaluations and Metrics
• English and Chinese Mathematical Reasoning: We conduct comprehensive assessments
of our models on English and Chinese benchmarks, covering mathematical problems
3


from grade-school level to college level. English benchmarks include GSM8K (Cobbe
et al., 2021), MATH (Hendrycks et al., 2021), SAT (Azerbayev et al., 2023), OCW Courses
(Lewkowycz et al., 2022a), MMLU-STEM (Hendrycks et al., 2020). Chinese benchmarks
include MGSM-zh (Shi et al., 2023), CMATH (Wei et al., 2023), Gaokao-MathCloze (Zhong
et al., 2023), and Gaokao-MathQA (Zhong et al., 2023). We evaluate models’ ability
to generate self-contained text solutions without tool use, and also the ability to solve
problems using Python.
On English benchmarks, DeepSeekMath-Base is competitive with the closed-source Min-
erva 540B (Lewkowycz et al., 2022a), and surpasses all open-source base models (e.g., Mis-
tral 7B (Jiang et al., 2023) and Llemma-34B (Azerbayev et al., 2023)), regardless of whether
they’ve undergone math pre-training or not, often by a significant margin. Notably,
DeepSeekMath-Base is superior on Chinese benchmarks, likely because we don’t follow
previous works (Azerbayev et al., 2023; Lewkowycz et al., 2022a) to collect English-only
math pre-training data, and also include high-quality non-English ones. With mathemati-
cal instruction tuning and reinforcement learning, the resulting DeepSeekMath-Instruct
and DeepSeekMath-RL demonstrate strong performance, obtaining an accuracy of over
50% on the competition-level MATH dataset for the first time within the open-source
community.
• Formal Mathematics: We evaluate DeepSeekMath-Base using the informal-to-formal
theorem proving task from (Jiang et al., 2022) on miniF2F (Zheng et al., 2021) with Isabelle
(Wenzel et al., 2008) chosen to be the proof assistant. DeepSeekMath-Base demonstrates
strong few-shot autoformalization performance.
• Natural Language Understanding, Reasoning, and Code: To build a comprehensive
profile of models’ general understanding, reasoning, and coding capabilities, we eval-
uate DeepSeekMath-Base on the Massive Multitask Language Understanding (MMLU)
benchmark (Hendrycks et al., 2020) which encompasses 57 multiple-choice tasks covering
diverse subjects, BIG-Bench Hard (BBH) (Suzgun et al., 2022) which consists of 23 chal-
lenging tasks that mostly require multi-step reasoning to solve, as well as HumanEval
(Chen et al., 2021) and MBPP (Austin et al., 2021) which are widely used to evaluate code
language models. Math pre-training benefits both language understanding and reasoning
performance.
2. Math Pre-Training
2.1. Data Collection and Decontamination
In this section, we will outline the process of constructing the DeepSeekMath Corpus from
Common Crawl. As depicted in Figure 2, we present an iterative pipeline that demonstrates
how to systematically gather a large-scale mathematical corpus from Common Crawl, starting
with a seed corpus (e.g., a small but high-quality collection of math-related dataset). It’s worth
noting that this approach is also applicable to other domains, such as coding.
First, we choose OpenWebMath (Paster et al., 2023), a collection of high-quality mathematical
web texts, as our initial seed corpus. Using this corpus, we train a fastText model (Joulin et al.,
2016) to recall more OpenWebMath-like mathematical web pages. Specifically, we randomly
select 500,000 data points from the seed corpus as positive training examples and another
500,000 web pages from Common Crawl as negative ones. We employ an open-source library1
for training, configuring the vector dimension to 256, learning rate to 0.1, the maximum length
1https://fasttext.cc
4


Math Seed
Math Corpus
1. Train a FastText Model
2. Recall Math-Related Webpages 
From Common Crawl
3. Discover Math-Related Domains
4. Annotate Math-Related 
URL Path From Labelers
Deduplicated Common Crawl
40B HTML pages
Figure 2 | An iterative pipeline that collects mathematical web pages from Common Crawl.
of word n-gram to 3, the minimum number of word occurrences to 3, and the number of
training epochs to 3. To reduce the size of the original Common Crawl, we employ URL-based
deduplication and near-deduplication techniques, resulting in 40B HTML web pages. We then
recall mathematical web pages from deduplicated Common Crawl with the fastText model.
To filter out low-quality mathematical content, we rank the collected pages according to their
scores predicted by the fastText model, and only preserve the top-ranking ones. The volume
of data preserved is assessed through pre-training experiments on the top 40B, 80B, 120B, and
160B tokens. In the first iteration, we choose to keep the top 40B tokens.
After the first iteration of data collection, numerous mathematical web pages remain un-
collected, mainly because the fastText model is trained on a set of positive examples that lacks
sufficient diversity. We therefore identify additional mathematical web sources to enrich the seed
corpus, so that we can optimize the fastText model. Specifically, we first organize the entire Com-
mon Crawl into disjoint domains; a domain is defined as web pages sharing the same base URL.
For each domain, we calculate the percentage of web pages that are collected in the first iteration.
Domains where over 10% of the web pages have been collected are classified as math-related
(e.g., mathoverflow.net). Subsequently, we manually annotate the URLs associated with
mathematical content within these identified domains (e.g., mathoverflow.net/questions).
Web pages linked to these URLs, yet uncollected, will be added to the seed corpus. This ap-
proach enables us to gather more positive examples, thereby training an improved fastText
model capable of recalling more mathematical data in the subsequent iteration. After four
iterations of data collection, we end up with 35.5M mathematical web pages, totaling 120B
tokens. In the fourth iteration, we notice that nearly 98% of the data has already been collected
in the third iteration, so we decide to cease data collection.
To avoid benchmark contamination, we follow Guo et al. (2024) to filter out web pages
containing questions or answers from English mathematical benchmarks such as GSM8K (Cobbe
et al., 2021) and MATH (Hendrycks et al., 2021) and Chinese benchmarks such as CMATH
(Wei et al., 2023) and AGIEval (Zhong et al., 2023). The filtering criteria are as follows: any
text segment containing a 10-gram string that matches exactly with any sub-string from the
evaluation benchmarks is removed from our math training corpus. For benchmark texts that
are shorter than 10 grams but have at least 3 grams, we employ exact matching to filter out
contaminated web pages.
5


2.2. Validating the Quality of the DeepSeekMath Corpus
We run pre-training experiments to investigate how the DeepSeekMath Corpus is compared
with the recently released math-training corpora:
• MathPile (Wang et al., 2023c): a multi-source corpus (8.9B tokens) aggregated from
textbooks, Wikipedia, ProofWiki, CommonCrawl, StackExchange, and arXiv, with the
majority (over 85%) sourced from arXiv;
• OpenWebMath (Paster et al., 2023): CommonCrawl data filtered for mathematical content,
totaling 13.6B tokens;
• Proof-Pile-2 (Azerbayev et al., 2023): a mathematical corpus consisting of OpenWeb-
Math, AlgebraicStack (10.3B tokens of mathematical code), and arXiv papers (28.0B to-
kens). When experimenting on Proof-Pile-2, we follow Azerbayev et al. (2023) to use an
arXiv:Web:Code ratio of 2:4:1.
2.2.1. Training Setting
We apply math training to a general pre-trained language model with 1.3B parameters, which
shares the same framework as the DeepSeek LLMs (DeepSeek-AI, 2024), denoted as DeepSeek-
LLM 1.3B. We separately train a model on each mathematical corpus for 150B tokens. All
experiments are conducted using the efficient and light-weight HAI-LLM (High-flyer, 2023)
training framework. Following the training practice of DeepSeek LLMs, we use the AdamW
optimizer (Loshchilov and Hutter, 2017) with 𝛽1 = 0.9, 𝛽2 = 0.95, and weight_decay = 0.1, along
with a multi-step learning rate schedule where the learning rate reaches the peak after 2,000
warmup steps, decreases to its 31.6% after 80% of the training process, and further decreases to
10.0% of the peak after 90% of the training process. We set the maximum value of learning rate
to 5.3e-4, and use a batch size of 4M tokens with a 4K context length.
Math Corpus
Size
English Benchmarks
Chinese Benchmarks
GSM8K MATH OCW
SAT
MMLU
STEM
CMATH
Gaokao
MathCloze
Gaokao
MathQA
No Math Training
N/A
2.9%
3.0%
2.9% 15.6%
19.5%
12.3%
0.8%
17.9%
MathPile
8.9B
2.7%
3.3%
2.2% 12.5%
15.7%
1.2%
0.0%
2.8%
OpenWebMath
13.6B
11.5%
8.9%
3.7% 31.3%
29.6%
16.8%
0.0%
14.2%
Proof-Pile-2
51.9B
14.3%
11.2%
3.7% 43.8%
29.2%
19.9%
5.1%
11.7%
DeepSeekMath Corpus 120.2B
23.8%
13.6%
4.8% 56.3%
33.1%
41.5%
5.9%
23.6%
Table 1 | Performance of DeepSeek-LLM 1.3B trained on different mathematical corpora, evalu-
ated using few-shot chain-of-thought prompting. Corpus sizes are calculated using our tokenizer
with a vocabulary size of 100K.
2.2.2. Evaluation Results
The DeepSeekMath Corpus is of high quality, covers multilingual mathematical content, and
is the largest in size.
• High-quality: We evaluate downstream performance on 8 mathematical benchmarks using
few-shot chain-of-thought prompting Wei et al. (2022). As shown in Table 1, there is a clear
performance lead of the model trained on the DeepSeekMath Corpus. Figure 3 shows that
the model trained on the DeepSeekMath Corpus demonstrates better performance than
6


Figure 3 | Benchmark curves of DeepSeek-LLM 1.3B trained on different mathematical corpora.
Proof-Pile-2 at 50B tokens (1 full epoch of Proof-Pile-2), indicating the average quality of
DeepSeekMath Corpus is higher.
• Multilingual: The DeepSeekMath Corpus encompasses data in multiple languages, pre-
dominantly featuring English and Chinese as the two most represented languages. As
shown in Table 1, training on the DeepSeekMath Corpus enhances mathematical reasoning
performance in both English and Chinese. In contrast, existing mathematical corpora,
which are primarily English-centric, show limited improvement and may even hinder
performance in Chinese mathematical reasoning.
• Large-scale: The DeepSeekMath Corpus is several times larger than existing mathematical
corpora. As depicted in Figure 3, DeepSeek-LLM 1.3B, when trained on the DeepSeek-
Math Corpus, shows a steeper learning curve along with more lasting improvements. In
contrast, the baseline corpora are much smaller, and have already been repeated multiple
rounds during training, with the resulting model performance quickly reaching a plateau.
2.3. Training and Evaluating DeepSeekMath-Base 7B
In this section, we introduce DeepSeekMath-Base 7B, a base model with strong reasoning
abilities, especially in mathematics. Our model is initialized with DeepSeek-Coder-Base-v1.5 7B
7


(Guo et al., 2024) and trained for 500B tokens. The distribution of the data is as follows: 56%
is from the DeepSeekMath Corpus, 4% from AlgebraicStack, 10% from arXiv, 20% is Github
code, and the remaining 10% is natural language data from Common Crawl in both English and
Chinese. We mainly adopt the training setting specified in Section 2.2.1, except that we set the
maximum value of the learning rate to 4.2e-4 and use a batch size of 10M tokens.
We conduct a comprehensive assessment of the mathematical capabilities of DeepSeekMath-
Base 7B, focusing on its ability to produce self-contained mathematical solutions without relying
on external tools, solve mathematical problems using tools, and conduct formal theorem proving.
Beyond mathematics, we also provide a more general profile of the base model, including its
performance of natural language understanding, reasoning, and programming skills.
Mathematical Problem Solving with Step-by-Step Reasoning
We evaluate DeepSeekMath-
Base’s performance of solving mathematical problems using few-shot chain-of-thought prompt-
ing (Wei et al., 2022), across eight benchmarks in English and Chinese. These benchmarks encom-
pass quantitative reasoning (e.g., GSM8K (Cobbe et al., 2021), MATH (Hendrycks et al., 2021),
and CMATH (Wei et al., 2023)) and multiple-choice problems (e.g., MMLU-STEM (Hendrycks
et al., 2020) and Gaokao-MathQA (Zhong et al., 2023)), covering diverse fields of mathematics
from elementary to college-level complexity.
As shown in Table 2, DeepSeekMath-Base 7B leads in performance across all eight bench-
marks among the open-source base models (including the widely-used general model Mistral
7B (Jiang et al., 2023) and the recently released Llemma 34B (Azerbayev et al., 2023) which
underwent math training on Proof-Pile-2 (Azerbayev et al., 2023)). Notably, on the competition-
level MATH dataset, DeepSeekMath-Base surpasses existing open-source base models by over
10% absolute, and outperforms Minerva 540B (Lewkowycz et al., 2022a), a closed-source base
model 77 times larger which builds on PaLM (Lewkowycz et al., 2022b) and is further trained
on mathematical texts.
Model
Size
English Benchmarks
Chinese Benchmarks
GSM8K MATH OCW
SAT
MMLU
STEM
CMATH
Gaokao
MathCloze
Gaokao
MathQA
Closed-Source Base Model
Minerva
7B
16.2%
14.1%
7.7%
-
35.6%
-
-
-
Minerva
62B
52.4%
27.6%
12.0%
-
53.9%
-
-
-
Minerva
540B
58.8%
33.6%
17.6%
-
63.9%
-
-
-
Open-Source Base Model
Mistral
7B
40.3%
14.3%
9.2%
71.9%
51.1%
44.9%
5.1%
23.4%
Llemma
7B
37.4%
18.1%
6.3%
59.4% 43.1%
43.4%
11.9%
23.6%
Llemma
34B
54.0%
25.3%
10.3% 71.9%
52.9%
56.1%
11.9%
26.2%
DeepSeekMath-Base 7B
64.2%
36.2% 15.4% 84.4%
56.5%
71.7%
20.3%
35.3%
Table 2 | Comparisons between DeepSeekMath-Base 7B and strong base models on English and
Chinese mathematical benchmarks. Models are evaluated with chain-of-thought prompting.
Minerva results are quoted from Lewkowycz et al. (2022a).
8


Mathematical Problem Solving with Tool Use
We evaluate program-aided mathematical
reasoning on GSM8K and MATH using few-shot program-of-thought prompting (Chen et al.,
2022; Gao et al., 2023). Models are prompted to solve each problem by writing a Python program
where libraries such as math and sympy can be utilized for intricate computations. The execution
result of the program is evaluated as the answer. As shown in Table 3, DeepSeekMath-Base 7B
outperforms the prior state-of-the-art Llemma 34B.
Model
Size
Problem Solving w/ Tools
Informal-to-Formal Proving
GSM8K+Python MATH+Python miniF2F-valid miniF2F-test
Mistral
7B
48.5%
18.2%
18.9%
18.0%
CodeLlama
7B
27.1%
17.2%
16.3%
17.6%
CodeLlama
34B
52.7%
23.5%
18.5%
18.0%
Llemma
7B
41.0%
18.6%
20.6%
22.1%
Llemma
34B
64.6%
26.3%
21.0%
21.3%
DeepSeekMath-Base 7B
66.9%
31.4%
25.8%
24.6%
Table 3 | Few-shot evaluation of base models’ ability to solve mathematical problems using tools
and the ability to conduct informal-to-formal theorem proving in Isabelle.
Formal Mathematics
Formal proof automation is beneficial to ensure the accuracy and relia-
bility of mathematical proofs and enhance efficiency, with increasing attention in recent years.
We evaluate DeepSeekMath-Base 7B on the task of informal-to-formal proving from (Jiang et al.,
2022) which is to generate a formal proof based on an informal statement, a formal counterpart
of the statement, and an informal proof. We evaluate on miniF2F (Zheng et al., 2021), a bench-
mark for formal Olympiad-level mathematics, and generate a formal proof in Isabelle for each
problem with few-shot prompting. Following Jiang et al. (2022), we leverage models to generate
proof sketches, and execute the off-the-shelf automated prover Sledgehammer (Paulson, 2010)
to fill in the missing details. As shown in Table 3, DeepSeekMath-Base 7B demonstrates strong
performance in proof autoformalization.
Model
Size MMLU
BBH
HumanEval (Pass@1) MBPP (Pass@1)
Mistral
7B
62.4%
55.7%
28.0%
41.4%
DeepSeek-Coder-Base-v1.5† 7B
42.9%
42.9%
40.2%
52.6%
DeepSeek-Coder-Base-v1.5
7B
49.1%
55.2%
43.2%
60.4%
DeepSeekMath-Base
7B
54.9%
59.5%
40.9%
52.6%
Table 4 | Evaluation on natural language understanding, reasoning, and code benchmarks.
DeepSeek-Coder-Base-v1.5† is the checkpoint right before learning rate decay, which is used to
train DeepSeekMath-Base. On MMLU and BBH, we use few-shot chain-of-thought prompting.
On HumanEval and MBPP, we evaluate model performance under the zero-shot setting and a
few-shot setting, respectively.
Natural Language Understanding, Reasoning, and Code
We evaluate model performance of
natural language understanding on MMLU (Hendrycks et al., 2020), reasoning on BBH (Suzgun
et al., 2022), and coding capabilities on HumanEval (Chen et al., 2021) and MBPP (Austin et al.,
9


2021). As shown in Table 4, DeepSeekMath-Base 7B exhibits significant enhancements in per-
formance on MMLU and BBH over its precursor, DeepSeek-Coder-Base-v1.5 (Guo et al., 2024),
illustrating the positive impact of math training on language understanding and reasoning.
Additionally, by including code tokens for continual training, DeepSeekMath-Base 7B effectively
maintains the performance of DeepSeek-Coder-Base-v1.5 on the two coding benchmarks. Over-
all, DeepSeekMath-Base 7B significantly outperforms the general model Mistral 7B (Jiang et al.,
2023) on the three reasoning and coding benchmarks.
3. Supervised Fine-Tuning
3.1. SFT Data Curation
We construct a mathematical instruction-tuning dataset covering English and Chinese problems
from different mathematical fields and of varying complexity levels: problems are paired with
solutions in chain-of-thought (CoT) (Wei et al., 2022), program-of-thought (PoT) (Chen et al.,
2022; Gao et al., 2023), and tool-integrated reasoning format (Gou et al., 2023). The total number
of training examples is 776K.
• English mathematical datasets: We annotate GSM8K and MATH problems with tool-
integrated solutions, and adopt a subset of MathInstruct (Yue et al., 2023) along with the
training set of Lila-OOD (Mishra et al., 2022) where problems are solved with CoT or
PoT. Our English collection covers diverse fields of mathematics, e.g., algebra, probability,
number theory, calculus, and geometry.
• Chinese mathematical datasets: We collect Chinese K-12 mathematical problems spanning
76 sub-topics such as linear equations, with solutions annotated in both CoT and tool-
integrated reasoning format.
3.2. Training and Evaluating DeepSeekMath-Instruct 7B
In this section, we introduce DeepSeekMath-Instruct 7B which undergoes mathematical instruc-
tion tuning based on DeepSeekMath-Base. Training examples are randomly concatenated until
reaching a maximum context length of 4K tokens. We train the model for 500 steps with a batch
size of 256 and a constant learning rate of 5e-5.
We evaluate models’ mathematical performance both without and with tool use, on 4
quantitative reasoning benchmarks in English and Chinese. We benchmark our model against
the leading models of the time:
• Closed-source models include: (1) the GPT family among which GPT-4 (OpenAI, 2023)
and GPT-4 Code Interpreter 2 are the most capable ones, (2) Gemini Ultra and Pro (Anil
et al., 2023), (3) Inflection-2 (Inflection AI, 2023), (4) Grok-1 3, as well as models recently
released by Chinese companies including (5) Baichuan-3 4, (6) the latest GLM-4 5 from the
GLM family (Du et al., 2022). These models are for general purposes, most of which have
undergone a series of alignment procedures.
• Open-source models include: general models like (1) DeepSeek-LLM-Chat 67B (DeepSeek-
AI, 2024), (2) Qwen 72B (Bai et al., 2023), (3) SeaLLM-v2 7B (Nguyen et al., 2023), and (4)
2https://openai.com/blog/chatgpt-plugins#code-interpreter
3https://x.ai/model-card
4https://www.baichuan-ai.com
5https://open.bigmodel.cn/dev/api#glm-4
10


ChatGLM3 6B (ChatGLM3 Team, 2023), as well as models with enhancements in mathemat-
ics including (5) InternLM2-Math 20B 6 which builds on InternLM2 and underwent math
training followed by instruction tuning, (6) Math-Shepherd-Mistral 7B which applys PPO
training (Schulman et al., 2017) to Mistral 7B (Jiang et al., 2023) with a process-supervised
reward model, (7) the WizardMath series (Luo et al., 2023) which improves mathematical
reasoning in Mistral 7B and Llama-2 70B (Touvron et al., 2023) using evolve-instruct (i.e.,
a version of instruction tuning that uses AI-evolved instructions) and PPO training with
training problems primarily sourced from GSM8K and MATH, (8) MetaMath 70B (Yu et al.,
2023) which is Llama-2 70B fine-tuned on an augmented version of GSM8K and MATH,
(9) ToRA 34B Gou et al. (2023) which is CodeLlama 34B fine-tuned to do tool-integrated
mathematical reasoning, (10) MAmmoTH 70B (Yue et al., 2023) which is Llama-2 70B
instruction-tuned on MathInstruct.
As shown in Table 5, under the evaluation setting where tool use is disallowed, DeepSeekMath-
Instruct 7B demonstrates strong performance of step-by-step reasoning.
Notably, on the
competition-level MATH dataset, our model surpasses all open-source models and the ma-
jority of proprietary models (e.g., Inflection-2 and Gemini Pro) by at least 9% absolute. This
is true even for models that are substantially larger (e.g., Qwen 72B) or have been specifi-
cally enhanced through math-focused reinforcement learning (e.g., WizardMath-v1.1 7B). While
DeepSeekMath-Instruct rivals the Chinese proprietary models GLM-4 and Baichuan-3 on MATH,
it still underperforms GPT-4 and Gemini Ultra.
Under the evaluation setting where models are allowed to integrate natural language rea-
soning and program-based tool use for problem solving, DeepSeekMath-Instruct 7B approaches
an accuracy of 60% on MATH, surpassing all existing open-source models. On the other bench-
marks, our model is competitive with DeepSeek-LLM-Chat 67B, the prior state-of-the-art that is
10 times larger.
4. Reinforcement Learning
4.1. Group Relative Policy Optimization
Reinforcement learning (RL) has been proven to be effective in further improving the mathe-
matical reasoning ability of LLMs after the Supervised Fine-Tuning (SFT) stage (Luo et al., 2023;
Wang et al., 2023b). In this section, we introduce our efficient and effective RL algorithm, Group
Relative Policy Optimization (GRPO).
4.1.1. From PPO to GRPO
Proximal Policy Optimization (PPO) (Schulman et al., 2017) is an actor-critic RL algorithm that is
widely used in the RL fine-tuning stage of LLMs (Ouyang et al., 2022). In particular, it optimizes
LLMs by maximizing the following surrogate objective:
J
𝑃𝑃𝑂(𝜃) = E[𝑞∼𝑃(𝑄), 𝑜∼𝜋𝜃𝑜𝑙𝑑(𝑂|𝑞)] 1
|𝑜|
|𝑜|
∑︁
𝑡=1
min
 𝜋𝜃(𝑜𝑡|𝑞, 𝑜<𝑡)
𝜋𝜃𝑜𝑙𝑑(𝑜𝑡|𝑞, 𝑜<𝑡) 𝐴𝑡, clip
 𝜋𝜃(𝑜𝑡|𝑞, 𝑜<𝑡)
𝜋𝜃𝑜𝑙𝑑(𝑜𝑡|𝑞, 𝑜<𝑡) , 1 −𝜀, 1 + 𝜀

𝐴𝑡

,
(1)
where 𝜋𝜃and 𝜋𝜃𝑜𝑙𝑑are the current and old policy models, and 𝑞, 𝑜are questions and outputs
sampled from the question dataset and the old policy 𝜋𝜃𝑜𝑙𝑑, respectively. 𝜀is a clipping-related
hyper-parameter introduced in PPO for stabilizing training. 𝐴𝑡is the advantage, which is
computed by applying Generalized Advantage Estimation (GAE) (Schulman et al., 2015), based
6https://github.com/InternLM/InternLM-Math
11


Model
Size English Benchmarks Chinese Benchmarks
GSM8K
MATH
MGSM-zh CMATH
Chain-of-Thought Reasoning
Closed-Source Model
Gemini Ultra
-
94.4%
53.2%
-
-
GPT-4
-
92.0%
52.9%
-
86.0%
Inflection-2
-
81.4%
34.8%
-
-
GPT-3.5
-
80.8%
34.1%
-
73.8%
Gemini Pro
-
86.5%
32.6%
-
-
Grok-1
-
62.9%
23.9%
-
-
Baichuan-3
-
88.2%
49.2%
-
-
GLM-4
-
87.6%
47.9%
-
-
Open-Source Model
InternLM2-Math
20B
82.6%
37.7%
-
-
Qwen
72B
78.9%
35.2%
-
-
Math-Shepherd-Mistral
7B
84.1%
33.0%
-
-
WizardMath-v1.1
7B
83.2%
33.0%
-
-
DeepSeek-LLM-Chat
67B
84.1%
32.6%
74.0%
80.3%
MetaMath
70B
82.3%
26.6%
66.4%
70.9%
SeaLLM-v2
7B
78.2%
27.5%
64.8%
-
ChatGLM3
6B
72.3%
25.7%
-
-
WizardMath-v1.0
70B
81.6%
22.7%
64.8%
65.4%
DeepSeekMath-Instruct 7B
82.9%
46.8%
73.2%
84.6%
DeepSeekMath-RL
7B
88.2%
51.7%
79.6%
88.8%
Tool-Integrated Reasoning
Closed-Source Model
GPT-4 Code Interpreter
-
97.0%
69.7%
-
-
Open-Source Model
InternLM2-Math
20B
80.7%
54.3%
-
-
DeepSeek-LLM-Chat
67B
86.7%
51.1%
76.4%
85.4%
ToRA
34B
80.7%
50.8%
41.2%
53.4%
MAmmoTH
70B
76.9%
41.8%
-
-
DeepSeekMath-Instruct 7B
83.7%
57.4%
72.0%
84.3%
DeepSeekMath-RL
7B
86.7%
58.8%
78.4%
87.6%
Table 5 | Performance of Open- and Closed-Source models with both Chain-of-Thought and
Tool-Integrated Reasoning on English and Chinese Benchmarks. Scores in gray denote majority
votes with 32 candidates; The others are Top1 scores. DeepSeekMath-RL 7B beats all open-
source models from 7B to 70B, as well as the majority of closed-source models. Although
DeepSeekMath-RL 7B is only further trained on chain-of-thought-format instruction tuning data
of GSM8K and MATH, it improves over DeepSeekMath-Instruct 7B on all benchmarks.
12


𝑞𝑞
𝑜𝑜!
𝑜𝑜"
𝑜𝑜#
𝑟𝑟
!
𝑟𝑟"
𝑟𝑟#
𝐴𝐴!
𝐴𝐴"
𝐴𝐴#
𝑞𝑞
𝑜𝑜
GAE
𝐴𝐴
𝑟𝑟
𝑣𝑣
Reward 
Model
Policy 
Model
Value 
Model
…
…
…
Policy 
Model
Reference 
Model
Reward 
Model
PPO
GRPO
Trained
Models
Frozen
Models
Reference 
Model
⊕
𝐾𝐾𝐾𝐾
𝐾𝐾𝐾𝐾
Group 
Computation
Figure 4 | Demonstration of PPO and our GRPO. GRPO foregoes the value model, instead
estimating the baseline from group scores, significantly reducing training resources.
on the rewards {𝑟≥𝑡} and a learned value function 𝑉𝜓. Thus, in PPO, a value function needs to
be trained alongside the policy model and to mitigate over-optimization of the reward model,
the standard approach is to add a per-token KL penalty from a reference model in the reward at
each token (Ouyang et al., 2022), i.e.,
𝑟𝑡= 𝑟𝜑(𝑞, 𝑜≤𝑡) −𝛽log 𝜋𝜃(𝑜𝑡|𝑞, 𝑜<𝑡)
𝜋𝑟𝑒𝑓(𝑜𝑡|𝑞, 𝑜<𝑡) ,
(2)
where 𝑟𝜑is the reward model, 𝜋𝑟𝑒𝑓is the reference model, which is usually the initial SFT model,
and 𝛽is the coefficient of the KL penalty.
As the value function employed in PPO is typically another model of comparable size as
the policy model, it brings a substantial memory and computational burden. Additionally,
during RL training, the value function is treated as a baseline in the calculation of the advantage
for variance reduction. While in the LLM context, usually only the last token is assigned a
reward score by the reward model, which may complicate the training of a value function that is
accurate at each token. To address this, as shown in Figure 4, we propose Group Relative Policy
Optimization (GRPO), which obviates the need for additional value function approximation as
in PPO, and instead uses the average reward of multiple sampled outputs, produced in response
to the same question, as the baseline. More specifically, for each question 𝑞, GRPO samples a
group of outputs {𝑜1, 𝑜2, · · · , 𝑜𝐺} from the old policy 𝜋𝜃𝑜𝑙𝑑and then optimizes the policy model
by maximizing the following objective:
J
𝐺𝑅𝑃𝑂(𝜃) = E[𝑞∼𝑃(𝑄), {𝑜𝑖}𝐺
𝑖=1 ∼𝜋𝜃𝑜𝑙𝑑(𝑂|𝑞)]
1
𝐺
𝐺
∑︁
𝑖=1
1
|𝑜𝑖|
|𝑜𝑖|
∑︁
𝑡=1

min
 𝜋𝜃(𝑜𝑖,𝑡|𝑞, 𝑜𝑖,<𝑡)
𝜋𝜃𝑜𝑙𝑑(𝑜𝑖,𝑡|𝑞, 𝑜𝑖,<𝑡)
ˆ
𝐴𝑖,𝑡, clip
 𝜋𝜃(𝑜𝑖,𝑡|𝑞, 𝑜𝑖,<𝑡)
𝜋𝜃𝑜𝑙𝑑(𝑜𝑖,𝑡|𝑞, 𝑜𝑖,<𝑡) , 1 −𝜀, 1 + 𝜀

ˆ
𝐴𝑖,𝑡

−𝛽D𝐾𝐿

𝜋𝜃||𝜋𝑟𝑒𝑓

,
(3)
where 𝜀and 𝛽are hyper-parameters, and ˆ
𝐴𝑖,𝑡is the advantage calculated based on relative
rewards of the outputs inside each group only, which will be detailed in the following subsec-
tions. The group relative way that GRPO leverages to calculate the advantages, aligns well with
the comparative nature of rewards models, as reward models are typically trained on datasets
of comparisons between outputs on the same question. Also note that, instead of adding KL
penalty in the reward, GRPO regularizes by directly adding the KL divergence between the
trained policy and the reference policy to the loss, avoiding complicating the calculation of ˆ
𝐴𝑖,𝑡.
13


Algorithm 1 Iterative Group Relative Policy Optimization
Input initial policy model 𝜋𝜃init; reward models 𝑟𝜑; task prompts D; hyperparameters 𝜀, 𝛽, 𝜇
1: policy model 𝜋𝜃←𝜋𝜃init
2: for iteration = 1, ..., I do
3:
reference model 𝜋𝑟𝑒𝑓←𝜋𝜃
4:
for step = 1, ..., M do
5:
Sample a batch D𝑏from D
6:
Update the old policy model 𝜋𝜃𝑜𝑙𝑑←𝜋𝜃
7:
Sample 𝐺outputs {𝑜𝑖}𝐺
𝑖=1 ∼𝜋𝜃𝑜𝑙𝑑(· | 𝑞) for each question 𝑞∈D𝑏
8:
Compute rewards {𝑟𝑖}𝐺
𝑖=1 for each sampled output 𝑜𝑖by running 𝑟𝜑
9:
Compute ˆ
𝐴𝑖,𝑡for the 𝑡-th token of 𝑜𝑖through group relative advantage estimation.
10:
for GRPO iteration = 1, ..., 𝜇do
11:
Update the policy model 𝜋𝜃by maximizing the GRPO objective (Equation 21)
12:
Update 𝑟𝜑through continuous training using a replay mechanism.
Output 𝜋𝜃
And different from the KL penalty term used in (2), we estimate the KL divergence with the
following unbiased estimator (Schulman, 2020):
D𝐾𝐿

𝜋𝜃||𝜋𝑟𝑒𝑓

=
𝜋𝑟𝑒𝑓(𝑜𝑖,𝑡|𝑞, 𝑜𝑖,<𝑡)
𝜋𝜃(𝑜𝑖,𝑡|𝑞, 𝑜𝑖,<𝑡) −log
𝜋𝑟𝑒𝑓(𝑜𝑖,𝑡|𝑞, 𝑜𝑖,<𝑡)
𝜋𝜃(𝑜𝑖,𝑡|𝑞, 𝑜𝑖,<𝑡) −1,
(4)
which is guaranteed to be positive.
4.1.2. Outcome Supervision RL with GRPO
Formally, for each question 𝑞, a group of outputs {𝑜1, 𝑜2, · · · , 𝑜𝐺} are sampled from the old
policy model 𝜋𝜃𝑜𝑙𝑑. A reward model is then used to score the outputs, yielding 𝐺rewards
r = {𝑟1, 𝑟2, · · · , 𝑟𝐺} correspondingly. Subsequently, these rewards are normalized by subtracting
the group average and dividing by the group standard deviation. Outcome supervision provides
the normalized reward at the end of each output 𝑜𝑖and sets the advantages ˆ
𝐴𝑖,𝑡of all tokens in
the output as the normalized reward, i.e., ˆ
𝐴𝑖,𝑡= e
𝑟𝑖= 𝑟𝑖−mean(r)
std(r)
, and then optimizes the policy by
maximizing the objective defined in equation (3).
4.1.3. Process Supervision RL with GRPO
Outcome supervision only provides a reward at the end of each output, which may not be
sufficient and efficient to supervise the policy in complex mathematical tasks. Following Wang
et al. (2023b), we also explore process supervision, which provides a reward at the end of
each reasoning step. Formally, given the question 𝑞and 𝐺sampled outputs {𝑜1, 𝑜2, · · · , 𝑜𝐺}, a
process reward model is used to score each step of the outputs, yielding corresponding rewards:
R = {{𝑟𝑖𝑛𝑑𝑒𝑥(1)
1
, · · · , 𝑟𝑖𝑛𝑑𝑒𝑥(𝐾1)
1
}, · · · , {𝑟𝑖𝑛𝑑𝑒𝑥(1)
𝐺
, · · · , 𝑟𝑖𝑛𝑑𝑒𝑥(𝐾𝐺)
𝐺
}}, where 𝑖𝑛𝑑𝑒𝑥( 𝑗) is the end token index
of the 𝑗-th step, and 𝐾𝑖is the total number of steps in the 𝑖-th output. We also normalize these
rewards with the average and the standard deviation, i.e.,e
𝑟𝑖𝑛𝑑𝑒𝑥( 𝑗)
𝑖
=
𝑟𝑖𝑛𝑑𝑒𝑥( 𝑗)
𝑖
−mean(R)
std(R)
. Subsequently,
the process supervision calculates the advantage of each token as the sum of the normalized
rewards from the following steps, i.e., ˆ
𝐴𝑖,𝑡= Í
𝑖𝑛𝑑𝑒𝑥( 𝑗)≥𝑡e
𝑟𝑖𝑛𝑑𝑒𝑥( 𝑗)
𝑖
, and then optimizes the policy by
maximizing the objective defined in equation (3).
14


4.1.4. Iterative RL with GRPO
As the reinforcement learning training process progresses, the old reward model may not be
sufficient to supervise the current policy model. Therefore, we also explore the iterative RL
with GRPO. As shown in Algorithm 1, in iterative GRPO, we generate new training sets for the
reward model based on the sampling results from the policy model and continually train the
old reward model using a replay mechanism that incorporates 10% of historical data. Then, we
set the reference model as the policy model, and continually train the policy model with the
new reward model.
4.2. Training and Evaluating DeepSeekMath-RL
We conduct RL based on DeepSeekMath-Instruct 7B. The training data of RL are chain-of-
thought-format questions related to GSM8K and MATH from the SFT data, which consists
of around 144K questions. We exclude other SFT questions to investigate the impact of RL
on benchmarks that lack data throughout the RL phase. We construct the training set of
reward models following (Wang et al., 2023b). We train our initial reward model based on the
DeepSeekMath-Base 7B with a learning rate of 2e-5. For GRPO, we set the learning rate of the
policy model as 1e-6. The KL coefficient is 0.04. For each question, we sample 64 outputs. The
max length is set to 1024, and the training batch size is 1024. The policy model only has a single
update following each exploration stage. We evaluate DeepSeekMath-RL 7B on benchmarks
following DeepSeekMath-Instruct 7B. For DeepSeekMath-RL 7B, GSM8K and MATH with
chain-of-thought reasoning can be regarded as in-domain tasks and all the other benchmarks
can be regarded as out-of-domain tasks.
Table 5 demonstrates the performance of open- and closed-source models with both chain-
of-thought and tool-integrated reasoning on English and Chinese benchmarks. We find that:
1) DeepSeekMath-RL 7B attains accuracies of 88.2% and 51.7% on GSM8K and MATH, respec-
tively, utilizing chain-of-thought reasoning. This performance surpasses that of all open-source
models in the 7B to 70B range, as well as the majority of closed-source models. 2) Crucially,
DeepSeekMath-RL 7B is only trained on chain-of-thought-format instruction tuning data of
GSM8K and MATH, starting from DeepSeekMath-Instruct 7B. Despite the constrained scope
of its training data, it outperforms DeepSeekMath-Instruct 7B across all evaluation metrics,
showcasing the effectiveness of reinforcement learning.
5. Discussion
In this section, we will share our findings in pre-training and RL experiments.
5.1. Lessons Learnt in Pre-Training
We first share our experience in pre-training. Unless otherwise specified, we will adhere to
the training settings outlined in Section 2.2.1. It is worth noting that, when referring to the
DeepSeekMath Corpus in this section, we use an 89B-token dataset from the second iteration of
the data collection process.
5.1.1. Code Training Benefits Mathematical Reasoning
A popular yet unverified hypothesis suggests that code training improves reasoning. We attempt
to offer a partial response to this, particularly within the mathematical domain: code training
15


Training Setting
Training Tokens
w/o Tool Use
w/ Tool Use
General Code Math GSM8K MATH CMATH GSM8K+Python MATH+Python
No Continual Training
–
–
–
2.9%
3.0%
12.3%
2.7%
2.3%
Two-Stage Training
Stage 1: General Training
400B
–
–
2.9%
3.2%
14.8%
3.3%
2.3%
Stage 2: Math Training
–
–
150B
19.1%
14.4%
37.2%
14.3%
6.7%
Stage 1: Code Training
–
400B –
5.9%
3.6%
19.9%
12.4%
10.0%
Stage 2: Math Training
–
–
150B
21.9%
15.3%
39.7%
17.4%
9.4%
One-Stage Training
Math Training
–
–
150B
20.5%
13.1%
37.6%
11.4%
6.5%
Code & Math Mixed Training –
400B 150B
17.6%
12.1%
36.3%
19.7%
13.5%
Table 6 | Investigation of how code affects mathematical reasoning under different training
settings. We experiment with DeepSeek-LLM 1.3B, and evaluate its mathematical reasoning
performance without and with tool use via few-shot chain-of-thought prompting and few-shot
program-of-thought prompting, respectively.
improves models’ ability to do mathematical reasoning both with and without tool use.
To study how code training affects mathematical reasoning, we experimented with the
following two-stage training and one-stage training settings:
Two-Stage Training
• Code Training for 400B Tokens →Math Training for 150B Tokens: We train DeepSeek-
LLM 1.3B for 400B code tokens followed by 150B math tokens;
• General Training for 400B Tokens →Math Training for 150B Tokens: As a control
experiment, we also experiment with general tokens (sampled from a large-scale general
corpus created by DeepSeek-AI) instead of code tokens in the first stage of training, in an
attempt to investigate the advantages of code tokens over general tokens in improving
mathematical reasoning.
One-Stage Training
• Math Training for 150B Tokens: We train DeepSeek-LLM 1.3B for 150B math tokens;
• Training on a mixture of 400B Code Tokens and 150B Math Tokens: Math training fol-
lowing code training degrades coding performance. We investigate whether code tokens,
when mixed with math tokens for one-stage training, would still improve mathematical
reasoning and also alleviate the problem of catastrophic forgetting.
Results
Table 6 and Table 7 demonstrate the downstream performance under different training
settings.
Code training benefits program-aided mathematical reasoning, both under the two-stage
training and one-stage training settings. As shown in Table 6, under the two-stage training
setting, code training alone already significantly enhances the ability to solve GSM8K and
MATH problems using Python. Math training in the second stage yields further improvements.
Interestingly, under the one-stage training setting, mixing code tokens and math tokens effec-
tively mitigates the issue of catastrophic forgetting that arises from two-stage training, and also
synergizes coding (Table 7) and program-aided mathematical reasoning (Table 6).
16


Training Setting
Training Tokens
MMLU
BBH
HumanEval (Pass@1) MBPP (Pass@1)
General Code Math
No Continual Training
–
–
–
24.5%
28.1%
12.2%
13.0%
Two-Stage Training
Stage 1: General Training
400B
–
–
25.9%
27.7%
15.2%
13.6%
Stage 2: Math Training
–
–
150B
33.1%
32.7%
12.8%
13.2%
Stage 1: Code Training
–
400B –
25.0%
31.5%
25.0%
40.0%
Stage 2: Math Training
–
–
150B
36.2%
35.3%
12.2%
17.0%
One-Stage Training
Math Training
–
–
150B
32.3%
32.5%
11.6%
13.2%
Code & Math Mixed Training –
400B 150B
33.5%
35.6%
29.3%
39.4%
Table 7 | Investigation of how different settings of code and math training affect model perfor-
mance of language understanding, reasoning, and coding. We experiment with DeepSeek-LLM
1.3B. We evaluate the models on MMLU and BBH using few-shot chain-of-thought prompting.
On HumanEval and MBPP, we conduct zero-shot and few-shot evaluations, respectively.
Model
Size ArXiv Corpus
English Benchmarks
Chinese Benchmarks
GSM8K MATH OCW
SAT
MMLU
STEM
CMATH
Gaokao
MathCloze
Gaokao
MathQA
DeepSeek-LLM
1.3B
No Math Training
2.9%
3.0%
2.9% 15.6%
19.5%
12.3%
0.8%
17.9%
MathPile
2.7%
3.3%
2.2% 12.5%
15.7%
1.2%
0.0%
2.8%
ArXiv-RedPajama
3.3%
3.4%
4.0%
9.4%
9.0%
7.4%
0.8%
2.3%
DeepSeek-Coder-Base-v1.5 7B
No Math Training
29.0%
12.5%
6.6% 40.6%
38.1%
45.9%
5.9%
21.1%
MathPile
23.6%
11.5%
7.0% 46.9%
35.8%
37.9%
4.2%
25.6%
ArXiv-RedPajama
28.1%
11.1%
7.7% 50.0%
35.2%
42.6%
7.6%
24.8%
Table 8 | Effect of math training on different arXiv datasets. Model performance is evaluated
with few-shot chain-of-thought prompting.
ArXiv Corpus
miniF2F-valid miniF2F-test
No Math Training
20.1%
21.7%
MathPile
16.8%
16.4%
ArXiv-RedPajama
14.8%
11.9%
Table 9 | Effect of math training on different arXiv corpora, the base model being DeepSeek-
Coder-Base-v1.5 7B. We evaluate informal-to-formal proving in Isabelle.
Code training also improves mathematical reasoning without tool use. Under the two-stage
training setting, the initial stage of code training already results in moderate enhancements.
It also boosts the efficiency of the subsequent math training, eventually leading to the best
performance. However, combining code tokens and math tokens for one-stage training com-
promises mathematical reasoning without tool use. One conjecture is that DeepSeek-LLM 1.3B,
due to its limited scale, lacks the capacity to fully assimilate both code and mathematical data
simultaneously.
5.1.2. ArXiv Papers Seem Ineffective in Improving Mathematical Reasoning
ArXiv papers are commonly included as a component of math pre-training data (Azerbayev
et al., 2023; Lewkowycz et al., 2022a; Polu and Sutskever, 2020; Wang et al., 2023c). However,
17


detailed analysis regarding their impact on mathematical reasoning has not been extensively
conducted. Perhaps counter-intuitively, according to our experiments, arXiv papers seem
ineffective in improving mathematical reasoning. We experiment with models of different sizes,
including DeepSeek-LLM 1.3B and DeepSeek-Coder-Base-v1.5 7B (Guo et al., 2024), using arXiv
corpora that underwent varied processing pipelines:
• MathPile (Wang et al., 2023c): an 8.9B-token corpus developed with cleaning and filtering
heuristic rules, over 85% of which are scientific arXiv papers;
• ArXiv-RedPajama (Computer, 2023): the entirety of arXiv LaTeX files with preambles,
comments, macros, and bibliographies removed, totaling 28.0B tokens.
In our experiments, we separately train DeepSeek-LLM 1.3B for 150B tokens and DeepSeek-
Coder-Base-v1.5 7B for 40B tokens on each arXiv corpus. It seems that arXiv papers are ineffective
in improving mathematical reasoning. When trained on a arXiv-only corpus, both models dis-
play no notable improvements or even deterioration across various mathematical benchmarks of
different complexities employed in this study. These benchmarks include quantitative reasoning
datasets like GSM8K and MATH (Table 8), multiple-choice challenges like MMLU-STEM (Table
8), and formal mathematics like miniF2F (Table 9).
However, this conclusion has its limitations and should be taken with a grain of salt. We
have not yet studied:
• The impact of arXiv tokens on specific math-related tasks not included in this research,
such as informalization of theorems which is to convert formal statements or proofs to
their informal versions;
• The effect of arXiv tokens when combined with other types of data;
• Whether the benefits of arXiv papers would manifest themselves at a larger model scale.
Thus, further exploration is required, which we leave for future studies.
5.2. Insights of Reinforcement Learning
5.2.1. Towards to a Unified Paradigm
In this section, we provide a unified paradigm to analyze different training methods, such as
SFT, RFT, DPO, PPO, GRPO, and further conduct experiments to explore the factors of the
unified paradigm. Generally, the gradient with respect to the parameter 𝜃of a training method
can be written as:
∇𝜃JA(𝜃) = E[(𝑞, 𝑜) ∼D
|       {z       }
𝐷𝑎𝑡𝑎𝑆𝑜𝑢𝑟𝑐𝑒
]
©
­
­
­
«
1
|𝑜|
|𝑜|
∑︁
𝑡=1
𝐺𝐶A(𝑞, 𝑜, 𝑡, 𝜋𝑟𝑓)
|               {z               }
𝐺𝑟𝑎𝑑𝑖𝑒𝑛𝑡𝐶𝑜𝑒𝑓𝑓𝑖𝑐𝑖𝑒𝑛𝑡
∇𝜃log 𝜋𝜃(𝑜𝑡|𝑞, 𝑜<𝑡)
ª
®
®
®
¬
.
(5)
There exist three key components: 1) Data Source D, which determines the training data; 2)
Reward Function 𝜋𝑟𝑓, which is the source of the training reward signal; 3) Algorithm A: which
processes the training data and the reward signal to the gradient coefficient 𝐺𝐶that determines
the magnitude of the penalty or reinforcement for the data. We analyze several representative
methods based on such a unified paradigm:
• Supervised Fine-tuning (SFT): SFT fine-tunes pretrained model on human selected SFT
data.
18


Methods
Data Source
Reward Function
Gradient Coefficient
SFT
𝑞, 𝑜∼𝑃𝑠𝑓𝑡(𝑄, 𝑂)
-
1
RFT
𝑞∼𝑃𝑠𝑓𝑡(𝑄), 𝑜∼𝜋𝑠𝑓𝑡(𝑂|𝑞)
Rule
Equation 10
DPO
𝑞∼𝑃𝑠𝑓𝑡(𝑄), 𝑜+, 𝑜−∼𝜋𝑠𝑓𝑡(𝑂|𝑞)
Rule
Equation 14
Online RFT
𝑞∼𝑃𝑠𝑓𝑡(𝑄), 𝑜∼𝜋𝜃(𝑂|𝑞)
Rule
Equation 10
PPO
𝑞∼𝑃𝑠𝑓𝑡(𝑄), 𝑜∼𝜋𝜃(𝑂|𝑞)
Model
Equation 18
GRPO
𝑞∼𝑃𝑠𝑓𝑡(𝑄), {𝑜𝑖}𝐺
𝑖=1 ∼𝜋𝜃(𝑂|𝑞)
Model
Equation 21
Table 10 | The data source and gradient coefficient of different methods. 𝑃𝑠𝑓𝑡denotes the data
distribution of supervised fine-tuning datasets. 𝜋𝜃𝑠𝑓𝑡and 𝜋𝜃denote the supervised fine-tuned
model and the real-time policy model during the online training process, respectively.
0
2000
4000
6000
8000
Steps
56
58
60
62
64
66
Acc (%)
GSM8K
0
2000
4000
6000
8000
Steps
27
28
29
30
Acc (%)
MATH
RFT
Online RFT
GRPO+OS
GRPO+PS
Figure 5 | Performance of the DeepSeekMath-Instruct 1.3B model, which was further trained
using various methods, on two benchmarks.
• Rejection Sampling Fine-tuning (RFT): RFT further fine-tunes the SFT model on the
filtered outputs sampled from the SFT model based on SFT questions. RFT filters the
outputs based on the correctness of their answers.
• Direct Preference Optimization (DPO): DPO further refines the SFT model by fine-tuning
it on augmented outputs sampled from the SFT model, using pair-wise DPO loss.
• Online Rejection Sampling Fine-tuning (Online RFT): Different from RFT, Online RFT
initiates the policy model using the SFT model and refines it by fine-tuning with the
augmented outputs sampled from the real-time policy model.
• PPO/GRPO: PPO/GRPO initializes the policy model using the SFT model and reinforces
it with the outputs sampled from the real-time policy model.
We summarize the components of these methods in Table 10. Please refer to Appendix A.1 for a
more detailed derivation process.
Observation about Data Source
We divide the data source into two categories, online sam-
pling, and offline sampling. Online sampling denotes that the training data is from the explo-
ration results of the real-time training policy model, while offline sampling denotes that the
19


0
1300
2300
3300
4300
5300
Steps
83
84
85
86
87
88
89
Acc (%)
GSM8K
0
1300
2300
3300
4300
5300
Steps
47
48
49
50
51
52
Acc (%)
MATH
Iteration-0
Iteration-1
Iteration-2
Figure 6 | Performance of iterative reinforcement learning with DeepSeekMath-Instruct 7B on
two benchmarks.
training data is from the sampling results of the initial SFT model. RFT and DPO follow the
offline style, while Online RFT and GRPO follow the online style.
As shown in Figure 5, we find that the Online RFT significantly outperforms RFT on two
benchmarks. Specifically, Online RFT is comparable to RFT in the early stage of training but
gains an absolute advantage in the later stage, demonstrating the superiority of online training.
This is intuitive, as in the initial stage, the actor and the SFT model exhibit close resemblance,
with the sampled data revealing only minor differences. In the later stage, however, the data
sampled from the actor will exhibit more significant differences, and real-time data sampling
will offer greater advantages.
Observation about Gradient Coefficient
The algorithm processes the reward signal to the
gradient coefficient to update the model parameter. We divide the reward function as ‘Rule’
and ‘Model’ in our experiments. Rule refers to judging the quality of a response based on
the correctness of the answer, and Model denotes that we train a reward model to score each
response. The training data of the reward model is based on the rule judgment. Equations 10
and 21 highlight a key difference between GRPO and Online RFT: GRPO uniquely adjusts its
gradient coefficient based on the reward value provided by the reward model. This allows for
differential reinforcement and penalization of responses according to their varying magnitudes.
In contrast, Online RFT lacks this feature; it does not penalize incorrect responses and uniformly
reinforces all responses with correct answers at the same level of intensity.
As demonstrated in Figure 5, GRPO surpasses online RFT, thereby highlighting the efficiency
of altering positive and negative gradient coefficients. In addition, GRPO+PS shows superior
performance compared to GRPO+OS, indicating the benefits of using fine-grained, step-aware
gradient coefficients. Furthermore, we explore the iterative RL, in our experiments, we conduct
two rounds of iteration. As shown in Figure 6, we notice that the iterative RL significantly
improves the performance, especially at the first iteration.
20


1
4
8
16
32
64
K: The number of candidates
82
84
86
88
90
92
94
96
98
Acc (%)
GSM8K
1
4
8
16
32
64
K: The number of candidates
45
50
55
60
65
70
75
80
85
Acc (%)
MATH
Maj@K-Instruct
Maj@K-RL
Pass@K-Instruct
Pass@K-RL
Figure 7 | The Maj@K and Pass@K of SFT and RL DeepSeekMath 7B on GSM8K and MATH
(temperature 0.7). It was noted that RL enhances Maj@K but not Pass@K.
5.2.2. Why RL Works?
In this paper, we conduct reinforcement learning based on a subset of instruction tuning
data, and it achieves significant performance enhancement upon the instruction tuning model.
To further explain why reinforcement learning works. We evaluate the Pass@K and Maj@K
accuracy of the Instruct and RL models on two benchmarks. As shown in Figure 7, RL enhances
Maj@K’s performance but not Pass@K. These findings indicate that RL enhances the model’s
overall performance by rendering the output distribution more robust, in other words, it seems
that the improvement is attributed to boosting the correct response from TopK rather than
the enhancement of fundamental capabilities. Similarly, (Wang et al., 2023a) identified a
misalignment problem in reasoning tasks within the SFT model, showing that the reasoning
performance of SFT models can be improved through a series of preference alignment strategies
(Song et al., 2023; Wang et al., 2023a; Yuan et al., 2023b).
5.2.3. How to Achieve More Effective RL?
We demonstrate RL works pretty well in mathematical reasoning tasks. We also provide a unified
paradigm to understand different representative training methods. Within this paradigm, all
methods are conceptualized as either direct or simplified RL techniques. As summarized in
Equation 5, there exist three key components: Data Source, Algorithm, and Reward Function.
We provide some potential future directions about the three components.
Data Source
Data source is the raw material of all training methods. In the context of RL, we
specifically refer to the data source as the unlabeled questions with the outputs sampled from
the policy model. In this paper, we only use the questions from the instruction tuning stage and
a naive nucleus sampling to sample outputs. We think this is a potential reason that our RL
pipeline only improves the Maj@K performance. In the future, we will explore our RL pipeline
on out-of-distribution question prompts, in conjunction with advanced sampling (decoding)
strategies, like those based on tree-search methods (Yao et al., 2023). Also, the efficient inference
techniques (Kwon et al., 2023; Leviathan et al., 2023; Xia et al., 2023, 2024), which determines
21


the exploration efficiency of policy models, also play an exceedingly important role.
Algorithms
Algorithms process the data and reward signal to the gradient coefficient to update
the model parameter. Based on Equation 5, to some extent, all methods now fully TRUST the
signal of the reward function to increase or decrease the conditional probability of a certain
token. However, it is impossible to ensure the reward signal is always reliable, especially in
extremely complex tasks. For example, even the PRM800K datasets (Lightman et al., 2023),
which have been carefully annotated by well-trained annotators, still contain approximately 20%
of incorrectly annotations7. To this end, we will explore the reinforcement learning algorithm
that is robust against noisy reward signals. We believe such WEAK-TO-STRONG (Burns et al.,
2023) alignment methods will bring a fundamental change to the learning algorithms.
Reward Function
Reward function is the source of the training signal. In RL, the reward
function is usually the neural reward model. We think there exist three important directions for
reward models: 1) How to enhance the generalization ability of the reward model. The reward
model must be effectively generalized to handle out-of-distribution questions and advanced
decoding outputs; otherwise, reinforcement learning may merely stabilize the distribution of
LLMs rather than improve their fundamental capabilities; 2) How to reflect the uncertainty
of reward model. The uncertainty could potentially act as a linking bridge between the weak
reward model and the weak-to-strong learning algorithms; 3) How to efficiently build high-
quality process reward models that can provide fine-grained training signals for the reasoning
process (Lightman et al., 2023; Wang et al., 2023b).
6. Conclusion, Limitation, and Future Work
We present DeepSeekMath, which outperforms all open-source models on the competition-
level MATH benchmark and approaches the performance of closed models. DeepSeekMath is
initialized with DeepSeek-Coder-v1.5 7B and undergoes continual training for 500B tokens, with
a significant component of the training data being 120B math tokens sourced from Common
Crawl. Our extensive ablation study shows web pages offer significant potential for high-quality
mathematical data, while arXiv may not as beneficial as we expected. We introduce Group
Relative Policy Optimization (GRPO), a variant of Proximal Policy Optimization (PPO), which
can notably improve mathematical reasoning capabilities with less memory consumption. The
experiment results show that GRPO is effective even if DeepSeekMath-Instruct 7B has reached
a high score on benchmarks. We also provide a unified paradigm to understand a series of
methods and summarize several potential directions for more effective reinforcement learning.
Although DeepSeekMath achieves impressive scores on quantitative reasoning benchmarks,
its capability on geometry and theorem-proof are relatively weaker than closed models. For
instance, in our dry run, the model cannot handle problems related to triangles and ellipses,
which may indicate data selection bias in pre-training and fine-tuning. In addition, restricted
by the model scale, DeepSeekMath is worse than GPT-4 on few-shot capability. GPT-4 could
improve its performance with few-shot inputs, while DeepSeekMath shows similar performance
in zero-shot and few-shot evaluation. In the future, we will further improve our engineered
data selection pipeline to construct more high-quality pre-trained corpus. In addition, we will
explore the potential directions (Section 5.2.3) for more effective reinforcement learning of LLMs.
7https://github.com/openai/prm800k/issues/12#issuecomment-1728491852
22


References
R. Anil, S. Borgeaud, Y. Wu, J. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth,
K. Millican, D. Silver, S. Petrov, M. Johnson, I. Antonoglou, J. Schrittwieser, A. Glaese, J. Chen,
E. Pitler, T. P. Lillicrap, A. Lazaridou, O. Firat, J. Molloy, M. Isard, P. R. Barham, T. Hennigan,
B. Lee, F. Viola, M. Reynolds, Y. Xu, R. Doherty, E. Collins, C. Meyer, E. Rutherford, E. Moreira,
K. Ayoub, M. Goel, G. Tucker, E. Piqueras, M. Krikun, I. Barr, N. Savinov, I. Danihelka,
B. Roelofs, A. White, A. Andreassen, T. von Glehn, L. Yagati, M. Kazemi, L. Gonzalez,
M. Khalman, J. Sygnowski, and et al. Gemini: A family of highly capable multimodal
models. CoRR, abs/2312.11805, 2023. doi: 10.48550/ARXIV.2312.11805. URL https:
//doi.org/10.48550/arXiv.2312.11805.
J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry,
Q. Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732,
2021.
Z. Azerbayev, H. Schoelkopf, K. Paster, M. D. Santos, S. McAleer, A. Q. Jiang, J. Deng, S. Bider-
man, and S. Welleck. Llemma: An open language model for mathematics. arXiv preprint
arXiv:2310.10631, 2023.
J. Bai, S. Bai, Y. Chu, Z. Cui, K. Dang, X. Deng, Y. Fan, W. Ge, Y. Han, F. Huang, et al. Qwen
technical report. arXiv preprint arXiv:2309.16609, 2023.
C. Burns, P. Izmailov, J. H. Kirchner, B. Baker, L. Gao, L. Aschenbrenner, Y. Chen, A. Ecoffet,
M. Joglekar, J. Leike, et al. Weak-to-strong generalization: Eliciting strong capabilities with
weak supervision. arXiv preprint arXiv:2312.09390, 2023.
ChatGLM3 Team. Chatglm3 series: Open bilingual chat llms, 2023. URL https://github.c
om/THUDM/ChatGLM3.
M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda,
N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin,
B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet,
F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herbert-Voss, W. H. Guss,
A. Nichol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse,
A. N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford, M. Knight, M. Brundage,
M. Murati, K. Mayer, P. Welinder, B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, and
W. Zaremba. Evaluating large language models trained on code. CoRR, abs/2107.03374, 2021.
URL https://arxiv.org/abs/2107.03374.
W. Chen, X. Ma, X. Wang, and W. W. Cohen. Program of thoughts prompting: Disentangling
computation from reasoning for numerical reasoning tasks. CoRR, abs/2211.12588, 2022. doi:
10.48550/ARXIV.2211.12588. URL https://doi.org/10.48550/arXiv.2211.12588.
K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek,
J. Hilton, R. Nakano, et al. Training verifiers to solve math word problems. arXiv preprint
arXiv:2110.14168, 2021.
T. Computer. Redpajama: an open dataset for training large language models, Oct. 2023. URL
https://github.com/togethercomputer/RedPajama-Data.
DeepSeek-AI. Deepseek LLM: scaling open-source language models with longtermism. CoRR,
abs/2401.02954, 2024. doi: 10.48550/ARXIV.2401.02954. URL https://doi.org/10.485
50/arXiv.2401.02954.
23


Z. Du, Y. Qian, X. Liu, M. Ding, J. Qiu, Z. Yang, and J. Tang. Glm: General language model
pretraining with autoregressive blank infilling. In Proceedings of the 60th Annual Meeting
of the Association for Computational Linguistics (Volume 1: Long Papers), pages 320–335,
2022.
L. Gao, A. Madaan, S. Zhou, U. Alon, P. Liu, Y. Yang, J. Callan, and G. Neubig. PAL: program-
aided language models. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and
J. Scarlett, editors, International Conference on Machine Learning, ICML 2023, 23-29 July
2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research,
pages 10764–10799. PMLR, 2023. URL https://proceedings.mlr.press/v202/gao23f.
html.
Z. Gou, Z. Shao, Y. Gong, Y. Shen, Y. Yang, M. Huang, N. Duan, and W. Chen. Tora: A tool-
integrated reasoning agent for mathematical problem solving. CoRR, abs/2309.17452, 2023.
doi: 10.48550/ARXIV.2309.17452. URL https://doi.org/10.48550/arXiv.2309.1745
2.
D. Guo, Q. Zhu, D. Yang, Z. Xie, K. Dong, W. Zhang, G. Chen, X. Bi, Y. Wu, Y. K. Li, F. Luo,
Y. Xiong, and W. Liang. Deepseek-coder: When the large language model meets programming
– the rise of code intelligence, 2024.
D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Measuring
massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020.
D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt. Mea-
suring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874,
2021.
High-flyer. Hai-llm: 高效且轻量的大模型训练工具, 2023. URL https://www.high-flyer.c
n/en/blog/hai-llm.
Inflection AI. Inflection-2, 2023. URL https://inflection.ai/inflection-2.
A. Q. Jiang, S. Welleck, J. P. Zhou, W. Li, J. Liu, M. Jamnik, T. Lacroix, Y. Wu, and G. Lample. Draft,
sketch, and prove: Guiding formal theorem provers with informal proofs. arXiv preprint
arXiv:2210.12283, 2022.
A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. d. l. Casas, F. Bressand,
G. Lengyel, G. Lample, L. Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023.
A. Joulin, E. Grave, P. Bojanowski, M. Douze, H. Jégou, and T. Mikolov. Fasttext. zip: Compress-
ing text classification models. arXiv preprint arXiv:1612.03651, 2016.
W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, and I. Stoica.
Efficient memory management for large language model serving with pagedattention. In
Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Y. Leviathan, M. Kalman, and Y. Matias. Fast inference from transformers via speculative
decoding. In International Conference on Machine Learning, pages 19274–19286. PMLR,
2023.
A. Lewkowycz, A. Andreassen, D. Dohan, E. Dyer, H. Michalewski, V. Ramasesh, A. Slone,
C. Anil, I. Schlag, T. Gutman-Solo, et al. Solving quantitative reasoning problems with
language models. Advances in Neural Information Processing Systems, 35:3843–3857, 2022a.
24


A. Lewkowycz, A. Andreassen, D. Dohan, E. Dyer, H. Michalewski, V. V. Ramasesh, A. Slone,
C. Anil, I. Schlag, T. Gutman-Solo, Y. Wu, B. Neyshabur, G. Gur-Ari, and V. Misra. Solving
quantitative reasoning problems with language models. In S. Koyejo, S. Mohamed, A. Agarwal,
D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems
35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New
Orleans, LA, USA, November 28 - December 9, 2022, 2022b. URL http://papers.nips.
cc/paper_files/paper/2022/hash/18abbeef8cfe9203fdf9053c9c4fe191-Abstr
act-Conference.html.
H. Lightman, V. Kosaraju, Y. Burda, H. Edwards, B. Baker, T. Lee, J. Leike, J. Schulman,
I. Sutskever, and K. Cobbe. Let’s verify step by step. arXiv preprint arXiv:2305.20050, 2023.
I. Loshchilov and F. Hutter.
Decoupled weight decay regularization.
arXiv preprint
arXiv:1711.05101, 2017.
H. Luo, Q. Sun, C. Xu, P. Zhao, J. Lou, C. Tao, X. Geng, Q. Lin, S. Chen, and D. Zhang.
Wizardmath: Empowering mathematical reasoning for large language models via reinforced
evol-instruct. arXiv preprint arXiv:2308.09583, 2023.
S. Mishra, M. Finlayson, P. Lu, L. Tang, S. Welleck, C. Baral, T. Rajpurohit, O. Tafjord, A. Sab-
harwal, P. Clark, and A. Kalyan. LILA: A unified benchmark for mathematical reasoning.
In Y. Goldberg, Z. Kozareva, and Y. Zhang, editors, Proceedings of the 2022 Conference on
Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab
Emirates, December 7-11, 2022, pages 5807–5832. Association for Computational Linguistics,
2022. doi: 10.18653/V1/2022.EMNLP-MAIN.392. URL https://doi.org/10.18653/v1/
2022.emnlp-main.392.
X. Nguyen, W. Zhang, X. Li, M. M. Aljunied, Q. Tan, L. Cheng, G. Chen, Y. Deng, S. Yang,
C. Liu, H. Zhang, and L. Bing. Seallms - large language models for southeast asia. CoRR,
abs/2312.00738, 2023. doi: 10.48550/ARXIV.2312.00738. URL https://doi.org/10.485
50/arXiv.2312.00738.
OpenAI. GPT4 technical report. arXiv preprint arXiv:2303.08774, 2023.
L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal,
K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback.
Advances in Neural Information Processing Systems, 35:27730–27744, 2022.
K. Paster, M. D. Santos, Z. Azerbayev, and J. Ba. Openwebmath: An open dataset of high-quality
mathematical web text. CoRR, abs/2310.06786, 2023. doi: 10.48550/ARXIV.2310.06786. URL
https://doi.org/10.48550/arXiv.2310.06786.
L. C. Paulson. Three years of experience with sledgehammer, a practical link between auto-
matic and interactive theorem provers. In R. A. Schmidt, S. Schulz, and B. Konev, editors,
Proceedings of the 2nd Workshop on Practical Aspects of Automated Reasoning, PAAR-2010,
Edinburgh, Scotland, UK, July 14, 2010, volume 9 of EPiC Series in Computing, pages 1–10.
EasyChair, 2010. doi: 10.29007/TNFD. URL https://doi.org/10.29007/tnfd.
S. Polu and I. Sutskever. Generative language modeling for automated theorem proving. CoRR,
abs/2009.03393, 2020. URL https://arxiv.org/abs/2009.03393.
R. Rafailov, A. Sharma, E. Mitchell, S. Ermon, C. D. Manning, and C. Finn. Direct preference
optimization: Your language model is secretly a reward model. 2023.
25


J. Schulman. Approximating kl divergence, 2020. URL http://joschu.net/blog/kl-app
rox.html.
J. Schulman, P. Moritz, S. Levine, M. Jordan, and P. Abbeel. High-dimensional continuous
control using generalized advantage estimation. arXiv preprint arXiv:1506.02438, 2015.
J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization
algorithms. arXiv preprint arXiv:1707.06347, 2017.
F. Shi, M. Suzgun, M. Freitag, X. Wang, S. Srivats, S. Vosoughi, H. W. Chung, Y. Tay, S. Ruder,
D. Zhou, D. Das, and J. Wei. Language models are multilingual chain-of-thought reasoners.
In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali,
Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL https://openreview.net/pdf?id=
fR3wGCk-IXp.
F. Song, B. Yu, M. Li, H. Yu, F. Huang, Y. Li, and H. Wang. Preference ranking optimization for
human alignment. arXiv preprint arXiv:2306.17492, 2023.
M. Suzgun, N. Scales, N. Schärli, S. Gehrmann, Y. Tay, H. W. Chung, A. Chowdhery, Q. V. Le,
E. H. Chi, D. Zhou, et al. Challenging big-bench tasks and whether chain-of-thought can solve
them. arXiv preprint arXiv:2210.09261, 2022.
T. Tao. Embracing change and resetting expectations, 2023. URL https://unlocked.micro
soft.com/ai-anthology/terence-tao/.
H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra,
P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. Canton-Ferrer, M. Chen, G. Cucurull, D. Esiobu,
J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal, A. Hartshorn, S. Hosseini,
R. Hou, H. Inan, M. Kardas, V. Kerkez, M. Khabsa, I. Kloumann, A. Korenev, P. S. Koura,
M. Lachaux, T. Lavril, J. Lee, D. Liskovich, Y. Lu, Y. Mao, X. Martinet, T. Mihaylov, P. Mishra,
I. Molybog, Y. Nie, A. Poulton, J. Reizenstein, R. Rungta, K. Saladi, A. Schelten, R. Silva, E. M.
Smith, R. Subramanian, X. E. Tan, B. Tang, R. Taylor, A. Williams, J. X. Kuan, P. Xu, Z. Yan,
I. Zarov, Y. Zhang, A. Fan, M. Kambadur, S. Narang, A. Rodriguez, R. Stojnic, S. Edunov, and
T. Scialom. Llama 2: Open foundation and fine-tuned chat models. CoRR, abs/2307.09288,
2023. doi: 10.48550/arXiv.2307.09288. URL https://doi.org/10.48550/arXiv.2307.
09288.
T. H. Trinh, Y. Wu, Q. V. Le, H. He, and T. Luong. Solving olympiad geometry without human
demonstrations. Nature, 625(7995):476–482, 2024.
P. Wang, L. Li, L. Chen, F. Song, B. Lin, Y. Cao, T. Liu, and Z. Sui. Making large language models
better reasoners with alignment. arXiv preprint arXiv:2309.02144, 2023a.
P. Wang, L. Li, Z. Shao, R. Xu, D. Dai, Y. Li, D. Chen, Y. Wu, and Z. Sui. Math-shepherd: Verify
and reinforce llms step-by-step without human annotations. CoRR, abs/2312.08935, 2023b.
Z. Wang, R. Xia, and P. Liu. Generative AI for math: Part I - mathpile: A billion-token-scale
pretraining corpus for math. CoRR, abs/2312.17120, 2023c. doi: 10.48550/ARXIV.2312.17120.
URL https://doi.org/10.48550/arXiv.2312.17120.
J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. H. Chi, Q. V. Le, and D. Zhou.
Chain-of-thought prompting elicits reasoning in large language models. In NeurIPS, 2022.
URL http://papers.nips.cc/paper_files/paper/2022/hash/9d5609613524ecf
4f15af0f7b31abca4-Abstract-Conference.html.
26


T. Wei, J. Luan, W. Liu, S. Dong, and B. Wang. Cmath: Can your language model pass chinese
elementary school math test?, 2023.
M. Wenzel, L. C. Paulson, and T. Nipkow. The isabelle framework. In O. A. Mohamed, C. A.
Muñoz, and S. Tahar, editors, Theorem Proving in Higher Order Logics, 21st International
Conference, TPHOLs 2008, Montreal, Canada, August 18-21, 2008. Proceedings, volume 5170
of Lecture Notes in Computer Science, pages 33–38. Springer, 2008. doi: 10.1007/978-3-540-7
1067-7\_7. URL https://doi.org/10.1007/978-3-540-71067-7_7.
H. Xia, T. Ge, P. Wang, S.-Q. Chen, F. Wei, and Z. Sui. Speculative decoding: Exploiting
speculative execution for accelerating seq2seq generation. In H. Bouamor, J. Pino, and K. Bali,
editors, Findings of the Association for Computational Linguistics: EMNLP 2023, pages 3909–
3925, Singapore, Dec. 2023. Association for Computational Linguistics. doi: 10.18653/v1/20
23.findings-emnlp.257. URL https://aclanthology.org/2023.findings-emnlp.257.
H. Xia, Z. Yang, Q. Dong, P. Wang, Y. Li, T. Ge, T. Liu, W. Li, and Z. Sui. Unlocking efficiency
in large language model inference: A comprehensive survey of speculative decoding. arXiv
preprint arXiv:2401.07851, 2024.
S. Yao, D. Yu, J. Zhao, I. Shafran, T. L. Griffiths, Y. Cao, and K. Narasimhan. Tree of thoughts:
Deliberate problem solving with large language models. arXiv preprint arXiv:2305.10601,
2023.
L. Yu, W. Jiang, H. Shi, J. Yu, Z. Liu, Y. Zhang, J. T. Kwok, Z. Li, A. Weller, and W. Liu.
Metamath: Bootstrap your own mathematical questions for large language models. CoRR,
abs/2309.12284, 2023. doi: 10.48550/ARXIV.2309.12284. URL https://doi.org/10.485
50/arXiv.2309.12284.
Z. Yuan, H. Yuan, C. Li, G. Dong, C. Tan, and C. Zhou. Scaling relationship on learning
mathematical reasoning with large language models. arXiv preprint arXiv:2308.01825, 2023a.
Z. Yuan, H. Yuan, C. Tan, W. Wang, S. Huang, and F. Huang. Rrhf: Rank responses to align
language models with human feedback without tears. arXiv preprint arXiv:2304.05302, 2023b.
X. Yue, X. Qu, G. Zhang, Y. Fu, W. Huang, H. Sun, Y. Su, and W. Chen. Mammoth: Building
math generalist models through hybrid instruction tuning. CoRR, abs/2309.05653, 2023. doi:
10.48550/ARXIV.2309.05653. URL https://doi.org/10.48550/arXiv.2309.05653.
K. Zheng, J. M. Han, and S. Polu. Minif2f: a cross-system benchmark for formal olympiad-level
mathematics. arXiv preprint arXiv:2109.00110, 2021.
W. Zhong, R. Cui, Y. Guo, Y. Liang, S. Lu, Y. Wang, A. Saied, W. Chen, and N. Duan. AGIEval: A
human-centric benchmark for evaluating foundation models. CoRR, abs/2304.06364, 2023.
doi: 10.48550/arXiv.2304.06364. URL https://doi.org/10.48550/arXiv.2304.06364.
27


A. Appendix
A.1. Analysis of Reinforcement Learning
We provide the detailed derivation of the data source and gradient coefficient (algorithm and
reward function) across various methods, including SFT, RFT, Online RFT, DPO, PPO, and
GRPO.
A.1.1. Supervised Fine-tuning
The objective of Supervised Fine-tuning is maximizing the following objective:
J
𝑆𝐹𝑇(𝜃) = E[𝑞, 𝑜∼𝑃𝑠𝑓𝑡(𝑄, 𝑂)]
 
1
|𝑜|
|𝑜|
∑︁
𝑡=1
log 𝜋𝜃(𝑜𝑡|𝑞, 𝑜<𝑡)
!
.
(6)
The gradient of J
𝑆𝐹𝑇(𝜃) is:
∇𝜃J
𝑆𝐹𝑇= E[𝑞, 𝑜∼𝑃𝑠𝑓𝑡(𝑄, 𝑂)]
 
1
|𝑜|
|𝑜|
∑︁
𝑡=1
∇𝜃log 𝜋𝜃(𝑜𝑡|𝑞, 𝑜<𝑡)
!
.
(7)
Data Source: The dataset employed for SFT. Reward Function: This can be regarded as human
selection. Gradient Coefficient: always set to 1.
A.1.2. Rejection Sampling Fine-tuning
Rejection Sampling Fine-tuning first samples multiple outputs from the supervised fine-tuned
LLMs for each question, and then trains LLMs on the sampled outputs with the correct answer.
Formally, the objective of RFT is to maximize the following objectives:
J
𝑅𝐹𝑇(𝜃) = E[𝑞∼𝑃𝑠𝑓𝑡(𝑄), 𝑜∼𝜋𝑠𝑓𝑡(𝑂|𝑞)]
 
1
|𝑜|
|𝑜|
∑︁
𝑡=1
I(𝑜) log 𝜋𝜃(𝑜𝑡|𝑞, 𝑜<𝑡)
!
.
(8)
The gradient of J
𝑅𝐹𝑇(𝜃) is:
∇𝜃J
𝑅𝐹𝑇(𝜃) = E[𝑞∼𝑃𝑠𝑓𝑡(𝑄), 𝑜∼𝜋𝑠𝑓𝑡(𝑂|𝑞)]
 
1
|𝑜|
|𝑜|
∑︁
𝑡=1
I(𝑜)∇𝜃log 𝜋𝜃(𝑜𝑡|𝑞, 𝑜<𝑡)
!
.
(9)
Data Source: question in SFT dataset with outputs sampled from SFT model. Reward Function:
Rule (whether the answer is correct or not). Gradient Coefficient:
𝐺𝐶𝑅𝐹𝑇(𝑞, 𝑜, 𝑡) = I(𝑜) =
(
1
the answer of o is correct
0
the answer of o is incorrect
(10)
A.1.3. Online Rejection Sampling Fine-tuning
The only difference between RFT and Online RFT is that the outputs of Online RFT are sampled
from the real-time policy model 𝜋𝜃, rather than from the SFT model 𝜋𝜃𝑠𝑓𝑡. Therefore, the gradient
of online RFT is:
∇𝜃J
𝑂𝑛𝑅𝐹𝑇(𝜃) = E[𝑞∼𝑃𝑠𝑓𝑡(𝑄), 𝑜∼𝜋𝜃(𝑂|𝑞)]
 
1
|𝑜|
|𝑜|
∑︁
𝑡=1
I(𝑜)∇𝜃log 𝜋𝜃(𝑜𝑡|𝑞, 𝑜<𝑡)
!
.
(11)
28


A.1.4. Direct Preference Optimization (DPO)
The objective of DPO is:
J
𝐷𝑃𝑂(𝜃) = E[𝑞∼𝑃𝑠𝑓𝑡(𝑄), 𝑜+, 𝑜−∼𝜋𝑠𝑓𝑡(𝑂|𝑞)] log 𝜎©
­
«
𝛽1
|𝑜+|
|𝑜+|
∑︁
𝑡=1
log
𝜋𝜃(𝑜+
𝑡|𝑞, 𝑜+
<𝑡)
𝜋ref(𝑜+
𝑡|𝑞, 𝑜+
<𝑡) −𝛽1
|𝑜−|
|𝑜−|
∑︁
𝑡=1
log
𝜋𝜃(𝑜−
<𝑡|𝑞, 𝑜−
<𝑡)
𝜋ref(𝑜−
<𝑡|𝑞, 𝑜−
<𝑡)
ª
®
¬
(12)
The gradient of J
𝐷𝑃𝑂(𝜃) is:
∇𝜃J
𝐷𝑃𝑂(𝜃) = E[𝑞∼𝑃𝑠𝑓𝑡(𝑄), 𝑜+, 𝑜−∼𝜋𝑠𝑓𝑡(𝑂|𝑞)] ©
­
«
1
|𝑜+|
|𝑜+|
∑︁
𝑡=1
𝐺𝐶𝐷𝑃𝑂(𝑞, 𝑜, 𝑡)∇𝜃log 𝜋𝜃(𝑜+
𝑡|𝑞, 𝑜+
<𝑡)
−
1
|𝑜−|
|𝑜−|
∑︁
𝑡=1
𝐺𝐶𝐷𝑃𝑂(𝑞, 𝑜, 𝑡)∇𝜃log 𝜋𝜃(𝑜−
𝑡|𝑞, 𝑜−
<𝑡)ª
®
¬
(13)
Data Source: question in SFT dataset with outputs sampled from SFT model. Reward Function:
human preference in the general domain (can be ‘Rule’ in mathematical tasks). Gradient
Coefficient:
𝐺𝐶𝐷𝑃𝑂(𝑞, 𝑜, 𝑡) = 𝜎

𝛽log
𝜋𝜃(𝑜−
𝑡|𝑞, 𝑜−
<𝑡)
𝜋ref(𝑜−
𝑡|𝑞, 𝑜−
<𝑡) −𝛽log
𝜋𝜃(𝑜+
𝑡|𝑞, 𝑜+
<𝑡)
𝜋ref(𝑜+
𝑡|𝑞, 𝑜+
<𝑡)

(14)
A.1.5. Proximal Policy Optimization (PPO)
The objective of PPO is:
J
𝑃𝑃𝑂(𝜃) = E[𝑞∼𝑃𝑠𝑓𝑡(𝑄), 𝑜∼𝜋𝜃𝑜𝑙𝑑(𝑂|𝑞)] 1
|𝑜|
|𝑜|
∑︁
𝑡=1
min
 𝜋𝜃(𝑜𝑡|𝑞, 𝑜<𝑡)
𝜋𝜃𝑜𝑙𝑑(𝑜𝑡|𝑞, 𝑜<𝑡) 𝐴𝑡, clip
 𝜋𝜃(𝑜𝑡|𝑞, 𝑜<𝑡)
𝜋𝜃𝑜𝑙𝑑(𝑜𝑡|𝑞, 𝑜<𝑡) , 1 −𝜀, 1 + 𝜀

𝐴𝑡

.
(15)
To simplify the analysis, it is assumed that the model only has a single update following each
exploration stage, thereby ensuring that 𝜋𝜃𝑜𝑙𝑑= 𝜋𝜃. In this case, we can remove the min and clip
operation:
J
𝑃𝑃𝑂(𝜃) = E[𝑞∼𝑃𝑠𝑓𝑡(𝑄), 𝑜∼𝜋𝜃𝑜𝑙𝑑(𝑂|𝑞)] 1
|𝑜|
|𝑜|
∑︁
𝑡=1
𝜋𝜃(𝑜𝑡|𝑞, 𝑜<𝑡)
𝜋𝜃𝑜𝑙𝑑(𝑜𝑡|𝑞, 𝑜<𝑡) 𝐴𝑡.
(16)
The gradient of J
𝑃𝑃𝑂(𝜃) is:
∇𝜃J
𝑃𝑃𝑂(𝜃) = E[𝑞∼𝑃𝑠𝑓𝑡(𝑄), 𝑜∼𝜋𝜃𝑜𝑙𝑑(𝑂|𝑞)] 1
|𝑜|
|𝑜|
∑︁
𝑡=1
𝐴𝑡∇𝜃log 𝜋𝜃(𝑜𝑡|𝑞, 𝑜<𝑡)
(17)
Data Source: question in SFT dataset with outputs sampled from policy model. Reward Function:
reward model. Gradient Coefficient:
𝐺𝐶𝑃𝑃𝑂(𝑞, 𝑜, 𝑡, 𝜋𝜃𝑟𝑚) = 𝐴𝑡,
(18)
where 𝐴𝑡is the advantage, which is computed by applying Generalized Advantage Estimation
(GAE) (Schulman et al., 2015), based on the rewards {𝑟≥𝑡} and a learned value function 𝑉𝜓.
A.1.6. Group Relative Policy Optimization (GRPO)
The objective of GRPO is (assume 𝜋𝜃𝑜𝑙𝑑= 𝜋𝜃for simplified analysis):
J
𝐺𝑅𝑃𝑂(𝜃) = E[𝑞∼𝑃𝑠𝑓𝑡(𝑄), {𝑜𝑖}𝐺
𝑖=1 ∼𝜋𝜃𝑜𝑙𝑑(𝑂|𝑞)]
1
𝐺
𝐺
∑︁
𝑖=1
1
|𝑜𝑖|
|𝑜𝑖|
∑︁
𝑡=1
 𝜋𝜃(𝑜𝑖,𝑡|𝑞, 𝑜𝑖,<𝑡)
𝜋𝜃𝑜𝑙𝑑(𝑜𝑖,𝑡|𝑞, 𝑜𝑖,<𝑡)
ˆ
𝐴𝑖,𝑡−𝛽(
𝜋𝑟𝑒𝑓(𝑜𝑖,𝑡|𝑞, 𝑜𝑖,<𝑡)
𝜋𝜃(𝑜𝑖,𝑡|𝑞, 𝑜𝑖,<𝑡) −log
𝜋𝑟𝑒𝑓(𝑜𝑖,𝑡|𝑞, 𝑜𝑖,<𝑡)
𝜋𝜃(𝑜𝑖,𝑡|𝑞, 𝑜𝑖,<𝑡) −1)

.
(19)
29


The gradient of J
𝐺𝑅𝑃𝑂(𝜃) is:
∇𝜃J
𝐺𝑅𝑃𝑂(𝜃) = E[𝑞∼𝑃𝑠𝑓𝑡(𝑄), {𝑜𝑖}𝐺
𝑖=1 ∼𝜋𝜃𝑜𝑙𝑑(𝑂|𝑞)]
1
𝐺
𝐺
∑︁
𝑖=1
1
|𝑜𝑖|
|𝑜𝑖|
∑︁
𝑡=1

ˆ
𝐴𝑖,𝑡+ 𝛽
 𝜋𝑟𝑒𝑓(𝑜𝑖,𝑡|𝑜𝑖,<𝑡)
𝜋𝜃(𝑜𝑖,𝑡|𝑜𝑖,<𝑡) −1

∇𝜃log 𝜋𝜃(𝑜𝑖,𝑡|𝑞, 𝑜𝑖,<𝑡).
(20)
Data Source: question in SFT dataset with outputs sampled from policy model. Reward Function:
reward model. Gradient Coefficient:
𝐺𝐶𝐺𝑅𝑃𝑂(𝑞, 𝑜, 𝑡, 𝜋𝜃𝑟𝑚) = ˆ
𝐴𝑖,𝑡+ 𝛽
 𝜋𝑟𝑒𝑓(𝑜𝑖,𝑡|𝑜𝑖,<𝑡)
𝜋𝜃(𝑜𝑖,𝑡|𝑜𝑖,<𝑡) −1

,
(21)
where ˆ
𝐴𝑖,𝑡is computed based on the group reward scores.
30


MAmmoTH2: Scaling Instructions from the Web
♢Xiang Yue∗
, ♠Tuney Zheng∗
, ♠Ge Zhang∗
, ♠Wenhu Chen∗
♢Carnegie Mellon University, ♠University of Waterloo
xyue2@andrew.cmu.edu
wenhuchen@uwaterloo.ca
https://tiger-ai-lab.github.io/MAmmoTH2/
Abstract
Instruction tuning improves the reasoning abilities of large language models
(LLMs), with data quality and scalability being the crucial factors. Most instruction
tuning data come from human crowd-sourcing or GPT-4 distillation. We propose a
paradigm to efficiently harvest 10 million naturally existing instruction data from
the pre-training web corpus to enhance LLM reasoning. Our approach involves
(1) recalling relevant documents, (2) extracting instruction-response pairs, and (3)
refining the extracted pairs using open-source LLMs. Fine-tuning base LLMs on
this dataset, we build MAmmoTH2 models, which significantly boost performance on
reasoning benchmarks. Notably, MAmmoTH2-7B’s (Mistral) performance increases
from 11% to 36.7% on MATH and from 36% to 68.4% on GSM8K without training
on any in-domain data. Further training MAmmoTH2 on public instruction tuning
datasets yields MAmmoTH2-Plus, achieving state-of-the-art performance on several
reasoning and chatbot benchmarks. Our work demonstrates how to harvest large-
scale, high-quality instruction data without costly human annotation or GPT-4
distillation, providing a new paradigm for building better instruction tuning data.
25
22
72
32
61
57
35
48
85
36
73
75
29
45
85
37
65
63
34
47
86
38
72
74
20
30
40
50
60
70
80
90
100
TheoremQA
MATH
GSM8K
GPQA
MMLU-STEM
BBH
Mixtral-8x7B-Instruct
Qwen-1.5-110B
MAmmoTH2-7B-Plus (Ours)
MAmmoTH2-8x7B-Plus (Ours)
60
24
23
68.7
33.8
32.6
0
20
40
60
80
MBPP
AlpacaEval2 ArenaHard
Mixtral-8x7B-Instruct
MAmmoTH2-8x7B-Plus (Ours)
Reasoning Benchmarks
Additional Benchmarks
Figure 1: Overview of MAmmoTH2-Plus results. The MAmmoTH2-8x7B-Plus variant outperforms
Mixtral-Instruct on reasoning benchmarks, matching Qwen-1.5-110B with only 13B active parameters.
It also surpasses Mixtral-Instruct by around 10 points on general code and chatbot benchmarks.
1
Introduction
Reasoning is a fundamental aspect of human cognition and problem-solving (Clark et al., 2018;
Hendrycks et al., 2021a; Cobbe et al., 2021; Rein et al., 2023; Yue et al., 2023a). Proficiency in
∗All of the authors are core contributors to the project.
arXiv:2405.03548v4  [cs.CL]  23 May 2024


Extract
Web
Recall
WebInstruct: 10M instruction data from the web
Distillation from GPT-4
Human annotation
Seed Data 
Costly, usually small-scale
Synthetic
Annotated
Prone to bias & hallucinations
Diverse, high quality and large-scale
Raw Doc
Extracted QA
Annotators
Previous Methods
Our Method
Refine
Refined QA
Figure 2: Comparison between our dataset curation method and previous studies.
reasoning is essential for advancing scientific knowledge, developing new technologies, and making
informed decisions in various contexts. Recently, large language models (LLMs) (Brown et al., 2020;
Ouyang et al., 2022; Touvron et al., 2023a,b; Achiam et al., 2023; Team et al., 2023) have shown
remarkable progress in various NLP tasks. However, their ability to perform complex reasoning
tasks (Lin et al., 2024) in the domains of mathematics, science, and engineering is still limited.
Recent studies have extensively explored how to enhance base LLMs’ reasoning abilities. The two
main approaches are continued training and instruction tuning. Continued training trains LLMs on
large-scale filtered documents (Lewkowycz et al., 2022; Taylor et al., 2022; Azerbayev et al., 2023;
Shao et al., 2024; Ying et al., 2024). Instruction tuning seeks to employ supervised fine-tuning loss on,
usually small-scale, high-quality instruction-response pairs (Ouyang et al., 2022; Chung et al., 2024).
While human-annotated instruction datasets (Cobbe et al., 2021; Hendrycks et al., 2021b; Amini
et al., 2019) are often limited in scale, recent studies (Yu et al., 2023; Yue et al., 2023b; Toshniwal
et al., 2024; Li et al., 2024a; Tang et al., 2024) attempt to prompt GPT-4 with seed data to increase
the scalability. However, the synthesized instruction data becomes highly biased, not diverse, and
prone to a high degree of hallucination.
To address these limitations, we propose to discover naturally existing instruction data from the
web (Figure 2). We argue that the pre-training corpus (e.g., Common Crawl) already contains a vast
amount of high-quality instruction data for LLM reasoning. For example, the web corpus contains a
large amount of educational materials in the form of instruction-following pairs. These documents
range across various domains like math, science, engineering, and humanities. Such readily available
instruction data is not only diverse but also of high quality. However, such instruction data is highly
dispersed across the corpus, which makes it particularly challenging to discover.
In this paper, we aim to mine these instruction-response pairs from the web using a three-step pipeline.
(1) Recall step: We create a diverse seed dataset by crawling several quiz websites. We use this
seed data to train a fastText model (Joulin et al., 2016) and employ it to recall documents from
Common Crawl (Computer, 2023). GPT-4 is used to trim down the recalled documents by their root
URL. We obtain 18M documents through this step. (2) Extract step: We utilize open-source LLMs
like Mixtral (Jiang et al., 2024) to extract Q-A pairs from these documents, producing roughly 5M
candidate Q-A pairs. (3) Refine step: After extraction, we further employ Mixtral-8×7B (Jiang et al.,
2024) and Qwen-72B (Bai et al., 2023) to refine (Zheng et al., 2024b) these candidate Q-A pairs. This
refinement operation aims to remove unrelated content, fix formality, and add missing explanations
to the candidate Q-A pairs. This refinement operation is pivotal to maintaining the quality of the
mined Q-A pairs. Eventually, we harvest a total of 10M instruction-response pairs through these
steps. Unlike existing instruction-tuning dataset, our dataset WEBINSTRUCT is purely mined from
the Web without any human crowdsourcing or GPT-4 distillation.
We validate the effectiveness of WEBINSTRUCT by training MAmmoTH2 on various base models
(Figure 1), including Mistral-7B (Jiang et al., 2023), Llama3-8B (Meta, 2024), Mixtral-8×7B (Jiang
et al., 2024), and Yi-34B (Young et al., 2024). MAmmoTH2 significantly outperforms the base models
on seven held-out reasoning benchmarks: TheoremQA (Chen et al., 2023b), GSM8K (Cobbe et al.,
2021), MATH (Hendrycks et al., 2021b), ARC-C (Clark et al., 2018), MMLU-STEM (Hendrycks
et al., 2021b), GPQA (Rein et al., 2023), and BBH (Suzgun et al., 2022). MAmmoTH2-7B improves
Mistral-7B’s performance by an average of 14 absolute points, while MAmmoTH2-34B enhances Yi-
34B’s performance by an average of 5.8 absolute points. Notably, Mistral-7B’s MATH accuracy can
rise from 11.2% to 36.7% after training on WEBINSTRUCT. As our dataset contains no in-domain
data from our evaluation benchmarks, this highlights the models’ strong generalization ability.
We further enhance MAmmoTH2’s performance on code generation, math reasoning, and instruction-
following tasks by tuning it on open-source instruction datasets, including OpenHermes2.5 (Teknium,
2


2023), Code-Feedback (Zheng et al., 2024c), and Math-plus. The resulting model, MAmmoTH2-Plus,
excels on seven reasoning benchmarks and other general tasks.
MAmmoTH2-7B-Plus and
MAmmoTH2-8B-Plus achieve state-of-the-art performance on TheoremQA, ARC-C, MMLU-STEM,
GPQA, and BBH, and competitive results on MATH (45%) and GSM8K (85%). MAmmoTH2-Plus
also performs well on general tasks, with MAmmoTH2-7B-Plus showing promising results on Hu-
manEval and MBPP, and MAmmoTH2-8×7B leading the AlpacaEval 2.0 and Arena Hard leaderboards.
Interestingly, MAmmoTH2-8B-Plus and Llama-3-8B-Instruct, both tuned from Llama-3-base using
datasets of the same size (10M), provide an apple-to-apple comparison. The only distinction is that
Llama-3-8B-Instruct is trained on 10M human-annotated dataset while we do not require any human
annotation. MAmmoTH2-8B-Plus outperforms Llama-3-Instruct by 6 points on reasoning tasks while
matching its performance on general tasks, reflecting WEBINSTRUCT’s cost-effectiveness advantage.
MAmmoTH2-Plus consistently surpasses official instruction models like Mixtral-Instruct on chat
benchmarks. These results demonstrate the effectiveness of our approach to scale up instruction data
from the web and offer a new perspective for future instruction tuning studies.
2
WEBINSTRUCT
In this section, we outline the process of constructing WEBINSTRUCT. Specifically, we divide the
data collection pipeline into three stages: (1) relevant document recall from the web corpus, (2) Q-A
pair extraction from recalled document, and (3) Q-A pair refinement. The full pipeline is depicted
in Figure 3 and an example for extraction and refinement is provided in Figure 4.
2.1
Recall from Common Crawl
In contrast to previous math-centric approaches (Paster et al., 2023; Wang et al., 2023c; Shao et al.,
2024), we aim for broad coverage of disciplines such as math, science, engineering, etc. Therefore,
careful balancing of the seed data is necessary to ensure diversity. However, publicly available
training datasets are mostly limited to mathematics. To address this issue, we propose to crawl new
Fast-Text
LLM Extraction
LLM Refinement
1
2
Recall from Web
Q-A Pair Extraction
Pre-process
Fast-Text
CC
CC
Seed
Decontaminate
URL Filter
3 Q-A Pair Refinement
URL Select
Figure 3: Step 1: Recall relevant documents from Common Crawl. Step 2: Extracting Q-A pairs.
Step 3: Refine with the extracted Q-A pairs.
Topics Science\nAnatomy&Physiology\nAstronomy\nAstrophysics
\nBiology\nChemistry \n...Socratic Meta...Featured Answers
How do you simplify #((u^4v^3)/(u^2v^-1)^4)^0# and write it using only 
positive exponents?
Answer by NickTheTurtle (Apr 1, 2017)
Explanation:\nAnything raised to the \(0^{th}\) power is simply 1.
\n\nRelated Questions\n What is the quotient of powers property?
\n\n\n\n How do you simplify expressions using the quotient
rule?…\nImpact of this question\n1274 views around the world
#Apps\niOS\nAndroid\nLinks\n[Privacy](#)\n[Terms](#)\n[Help](#)
Raw Docs
Question: How do you simplify (𝑢! 𝑣"
𝑢# 𝑣$% !
⁄
)& and write it using
only positive exponents?
Answer: Explanation: Anything to the 0𝑡ℎpower is just simply 1.
Extracted QA
Question:
How do you simplify (𝑢! 𝑣"
𝑢# 𝑣$% !
⁄
)&and write it using only
positive exponents?
Answer:
To simplify the expression (𝑢! 𝑣"
𝑢# 𝑣$% !
⁄
)& and rewrite it
using only positive exponents, we start by evaluating the
expression from the innermost operation outward.
First, consider the exponent of zero on the entire fraction:
Any expression raised to the power of zero is equal to 1 (provided
the base is not zero). Therefore, (𝑢! 𝑣"
𝑢# 𝑣$% !
⁄
)& = 1
This simplification makes the other calculations unnecessary
because raising any non-zero expression to the power of zero will
always result in 1.
So, the simplified expression is simply: 1
Rewritten QA
Unformatted Text, Site Information, Ads
Formatted QA but lacking detailed solutions
Formatted QA augmented with detailed solutions
Figure 4: An illustrating example from WEBINSTRUCT for the extraction and refinement step.
3


exam problems from several educational websites. These sites contain diverse problems from various
disciplines, helping to ensure diversity. We crawled 100K seed data as positive training examples
and randomly selected 100K negative documents from CC (Computer, 2023) to train a fastText
model (Joulin et al., 2016). The trained fastText model is used to recall relevant documents. We
employ the open-source fastText library with a vector dimension of 256 to train the model for 3
epochs, with a learning rate of 0.1, a maximum n-gram length of 3, and a maximum number of
word occurrences of 3. We recalled 100B tokens using the trained fasttext model from an internal
CC. These raw web documents are further grouped by their domains (root URL) and only domains
with more than 1000 documents are retained. We extracted roughly 600K domains from the recalled
documents. We then prompt GPT-3.5 to scan through the domains and automatically select those
that might contain instruction data. Around 50K domains are further labeled as positive samples
by GPT-3.5. Note that all the recalled documents in the first round are not kept for further usage
in Q-A Pair Extraction and Refinement. Next, we sample documents from the selected domains as
positive examples, and documents from the non-selected domains and general Common Crawl as
negative examples to re-train an improved fastText classifier. The newly trained fastText classifier
is used to recall documents. We recalled 40B tokens using the newly trained fastText model. We
prompt GPT-4 to sift through the recalled domains again, ultimately leading to 18M raw documents,
primarily originating from the desired websites.
2.2
Q-A Pair Extraction
We observe that a significant number of naturally existing Q-A pairs are present in the 18M documents.
However, these Q-A pairs are interspersed with a high volume of noise such as ads, markups,
boilerplate, etc. Our preliminary training on these raw documents only yields limited gains.
First, we carefully pre-process the HTML to pre-extract useful content from the recalled documents.
This is mostly rule-based filtering to clean site information, ads, HTML boilerplate, etc. This step
significantly reduces the document length for the next stage. We then prompt Qwen-72B (Bai et al.,
2023) to identify the question and answer pairs from the preprocessed documents. Specifically, we
provide a few in-context examples to help the model understand what to extract. We also allow the
model to return void if no natural question-answer pairs exist. In this stage, only 30% of the recalled
documents were identified as containing naturally existing Q-A pairs, resulting in roughly 5M Q-A
pairs as our candidates for the next step. However, these candidates still contain a substantial amount
of unrelated content and formality issues. Besides that, a large portion of the extracted Q-A pairs also
lack explanations for how the answer is derived. Therefore, we propose to perform another round of
refinement to increase the data quality.
To avoid contamination, we follow previous work (Shao et al., 2024) and filter out web pages
containing questions or answers to all of our evaluation benchmarks. Specifically, we filter out all
web pages that contain n-grams (n = 10) string matches with either the questions or answers.
2.3
Q-A Pair Refinement
To further improve the extracted Q-A pair candidates, we propose refining them using LLMs. In this
step, we prompt Mixtral-22B×8 (Jiang et al., 2024) and Qwen-72B (Bai et al., 2023) to reformat
the extracted Q-A pairs. If the answer does not contain any explanation, these two LLMs will
attempt to complete the intermediate reasoning steps leading to the given answer. We adopt two
models to increase the diversity of our dataset. Eventually, we harvest 10M Q-A pairs as our final
instruction-tuning dataset WEBINSTRUCT.
2.4
Dataset Statistics
To better distinguish our dataset from the existing ones, we include a summarization table in Table 1.
It can be observed that most SFT datasets contain less than 1M samples but are of high quality.
XwinMath (Li et al., 2024a) is the largest dataset, scaling up to over 1M samples through GPT4
synthesis, while OpenMathInstruct (Toshniwal et al., 2024) has not been generated using GPT-4 but
instead uses Mixtral-8x7B Jiang et al. (2024). However, the seed data for both datasets is only based
on GSM and MATH, leading to narrow domain coverage. In contrast, continue-training (CT) datasets
are normally filtered from the web with much larger size, often exceeding 10B tokens and even rising
to 120B tokens. However, continued pre-training on these massive datasets can be not only expensive
4


Table 1: The list of existing supervise-fine-tuning (SFT) and continue-training (CT) datasets. SFT
datasets are primarily from academic NLP sources or synthesized by GPT-3.5/4 using seed data. CT
datasets are larger but nosier. Our dataset falls between these two types.
Dataset
#Pairs
Domain
Format
Dataset Source
FLAN V2 (Chung et al., 2024)
100K
General
SFT
NLP data + Human CoT
Self-Instruct (Wang et al., 2023b)
82K
General
SFT
Generated by GPT3
GPT4-Alpaca (Taori et al., 2023)
52K
General
SFT
Generated by GPT4
SuperNI (Wang et al., 2022)
96K
General
SFT
NLP Datasets
Tora (Gou et al., 2023)
16K
Math
SFT
GSM+MATH Synthesis by GPT4
WizardMath (Luo et al., 2023)
96K
Math
SFT
GSM+MATH Synthesis by GPT4
MathInstruct (Yue et al., 2023b)
262K
Math
SFT
Math datasets Synthesis by GPT4
MetaMathQA (Yu et al., 2023)
395K
Math
SFT
GSM+MATH Synthesis by GPT3.5
XwinMath (Li et al., 2024a)
1.4M
Math
SFT
GSM+MATH Synthesis by GPT4
OpenMathInstruct (Toshniwal et al., 2024)
1.8M
Math
SFT
GSM+MATH Synthesis by Mixtral
Dataset
#Tokens
Domain
Format
Dataset Source
OpenWebMath (Paster et al., 2023)
12B
Math
LM
Filtered from Web
MathPile (Wang et al., 2023c)
10B
Math
LM
Filtered from Web
Cosmopeida (Ben Allal et al., 2024)
25B
General
LM
Synthesized by Mixtral
MINERVA (Lewkowycz et al., 2022)
38B
Math
LM
Filtered from Web
Proof-Pile-2 (Azerbayev et al., 2023)
55B
Math
LM
OpenWebMath+Arxiv+Code
Galactica (Taylor et al., 2022)
106B
Math & Sci.
LM
Filtered from Web
DeepseekMath (Shao et al., 2024)
120B
Math
LM
Recalled from Web
WEBINSTRUCT
(10M) 5B
Math & Sci.
SFT
Recall and Extracted from Web
but also ineffective due to the high noise ratio. WEBINSTRUCT, with roughly 5B tokens, strikes a
good balance between scalability and quality. It approaches the scalability of common CT datasets
while maintaining high quality through the three-step construction pipeline. This makes our dataset
unique compared to other alternatives.
2.5
Additional Public Instruction Datasets
To further enhance the diversity and quality of our dataset, we fine-tune MAmmoTH2 on several open-
source instruction tuning datasets. These datasets are carefully selected based on their relevance to
different reasoning subjects. Additionally, we consider some chat datasets to balance reasoning ability
and general chat ability. The open-source datasets we incorporate are OpenHermes 2.5 (Teknium,
2023), Code-Feedback (Zheng et al., 2024c) and our Math-Plus, which is an augmented version of
MetaMathQA (395K) (Yu et al., 2023) and Orca-Math (200K) (Mitra et al., 2024). More details of
the public datasets can be found in Appendix A.
3
Experimental Setup
3.1
Training Setup
We unify all the samples in our instruction dataset to conform to the structure of a multi-turn instruction
tuning dataset. This standardization ensures that the fine-tuned models can process data consistently,
regardless of the original dataset formats. We select the open-source models Mistral 7B (Jiang et al.,
2023), Mixtral 8×7B (Jiang et al., 2024), Llama-3 8B (Meta, 2024), and Yi-34B (Young et al., 2024)
as our base models. We fine-tune these models to validate our WEBINSTRUCT at multiple scales
using the LLaMA-Factory (Zheng et al., 2024d) library. We use a learning rate of 5e-6 for Mistral 7B
and 1e-5 for Mixtral, Llama-3 8B, and Yi 34B. The global batch size is set to 512 with a maximum
sequence length of 4096. We employ a cosine scheduler with a 3% warm-up period for 2 epochs. To
efficiently train the models, we utilize DeepSpeed (Rasley et al., 2020) with the ZeRO-3 stage. All
the models are trained with 32 A100 GPUs.
3.2
Evaluation Datasets
To rigorously assess the capabilities of models in reasoning abilities across different domains, we
utilize several widely used datasets, GSM8K (Cobbe et al., 2021), MATH (Hendrycks et al., 2021b),
TheoremQA (Chen et al., 2023b), BIG-Bench Hard (BBH) (Suzgun et al., 2022), ARC-C (Clark
5


Table 2: Main results on reasoning datasets. Models without the ’-Instruct’ suffix refer to the released
base models. Results are taken from official papers or blogs when available; otherwise, we use our
own evaluation script. Underscored results represent the best baseline scores under the size constraint.
All models are inferred with few-shot CoT: TheoremQA (5-shot), MATH (4-shot), GSM8K (4-shot),
GPQA (5-shot), MMLU-STEM (5-shot), BBH (3-shot), and ARC-C (8-shot).
Model
TheoremQA
MATH
GSM8K
GPQA
MMLU-ST
BBH
ARC-C
AVG
GPT-4-Turbo-0409
48.4
69.2
94.5
46.2
76.5
86.7
93.6
73.6
Parameter Size between 20B and 110B
Qwen-1.5-110B
34.9
49.6
85.4
35.9
73.4
74.8
91.6
63.6
Qwen-1.5-72B
29.3
46.8
77.6
36.3
68.5
68.0
92.2
59.8
Deepseek-LM-67B
25.3
15.9
66.5
31.8
57.4
71.7
86.8
50.7
Yi-34B
23.2
15.9
67.9
29.7
62.6
66.4
89.5
50.7
Llemma-34B
21.1
25.0
71.9
29.2
54.7
48.4
69.5
45.7
Mixtral-8×7B
23.2
28.4
74.4
29.7
59.7
66.8
84.7
52.4
Mixtral-8×7B-Instruct
25.3
22.1
71.7
32.4
61.4
57.3
84.7
50.7
Intern-Math-20B
17.1
37.7
82.9
28.9
50.1
39.3
68.6
46.4
Trained only with WEBINSTRUCT (All evaluations are held-out)
MAmmoTH2-34B
30.4
35.0
75.6
31.8
64.5
68.0
90.0
56.4
∆over Yi
+7.2
+19.1
+7.7
+2.1
+2.9
+1.2
+0.5
+5.8
MAmmoTH2-8x7B
32.2
39.0
75.4
36.8
67.4
71.1
87.5
58.9
∆over Mixtral
+9.2
+10.6
+1.0
+7.1
+7.4
+3.3
+2.8
+6.5
Continue trained with additional instruction datasets (All held-out except MATH and GSM8K)
MAmmoTH2-8x7B-Plus
34.1
47.0
86.4
37.8
72.4
74.1
88.4
62.9
∆over Qwen-1.5-110B
-0.8
-2.6
+1.0
+1.5
-1.0
-0.7
-4.0
-0.7
Parameter Size = 7B or 8B
Deepseek-7B
15.7
6.4
17.4
25.7
43.1
42.8
47.8
28.4
Qwen-1.5-7B
14.2
13.3
54.1
26.7
45.4
45.2
75.6
39.2
Mistral-7B
19.2
11.2
36.2
24.7
50.1
55.7
74.2
38.8
Gemma-7B
21.5
24.3
46.4
25.7
53.3
57.4
72.5
43.0
Llemma-7B
17.2
18.0
36.4
23.2
45.2
44.9
50.5
33.6
WizardMath-7B-1.1
11.7
33.0
83.2
28.7
52.7
56.7
76.9
49.0
Abel-7B-002
19.3
29.5
83.2
30.3
29.7
32.7
72.5
42.5
Intern-Math-7B
13.2
34.6
78.1
22.7
41.1
48.1
59.8
42.5
Rho-1-Math-7B
21.0
31.0
66.9
29.2
53.1
57.7
72.7
47.3
Deepseek-Math-7B
25.3
34.0
64.2
29.2
56.4
59.5
67.8
48.0
Deepseek-Math-Instruct
23.7
44.3
82.9
31.8
59.3
55.4
70.1
52.5
Llama-3-8B
20.1
21.3
54.8
27.2
55.6
61.1
78.6
45.5
Llama-3-8B-Instruct
22.8
30.0
79.5
34.5
60.2
66.0
80.8
53.4
Trained only with WEBINSTRUCT (All evaluations are held-out)
MAmmoTH2-7B
29.0
36.7
68.4
32.4
62.4
58.6
81.7
52.8
∆over Mistral
+9.8
+25.5
+32.2
+7.7
+12.3
+2.9
+7.5
+14.0
MAmmoTH2-8B
32.2
35.8
70.4
35.2
64.2
62.1
82.2
54.3
∆over Llama3
+12.2
+14.5
+15.6
+8.0
+8.6
+1.0
+3.6
+8.8
Continue trained with additional instruction datasets (All held-out except MATH and GSM8K)
MAmmoTH2-7B-Plus
29.2
45.0
84.7
36.8
64.5
63.1
83.0
58.0
MAmmoTH2-8B-Plus
32.5
42.8
84.1
37.3
65.7
67.8
83.4
59.1
∆over best baseline
+7.2
+0.7
+1.5
+2.8
+5.5
+1.8
+2.6
+5.7
et al., 2018), GPQA (Rein et al., 2023), MMLU-STEM (Hendrycks et al., 2021a). These datasets
collectively enable a comprehensive assessment of language models’ reasoning prowess across a
spectrum of complexity and realism. The details of the evaluation datasets are in Appendix B.
We further evaluate the models on additional code generation tasks (including HumanEval (Chen
et al., 2021), MBPP (Austin et al., 2021) and their augmented version (Liu et al., 2024)), general
LLM benchmarks like MMLU (Hendrycks et al., 2021a) and its recent robust and challenging version
MMLU-Pro (TIGER-Lab, 2024). We also consider chat benchmarks like MT-Bench (Zheng et al.,
2024a), AlpacaEval 2.0 (Li et al., 2023), and Arena Hard (Li et al., 2024b) to demonstrate the
generalizability of WEBINSTRUCT and WEBINSTRUCT-PLUS on more general LLM benchmarks.
6


4
Main Results
4.1
Experimental Results on Reasoning Benchmarks
Table 2 presents our main results, with existing models partitioned into two tracks based on their
parameter size. For 7B parameter models, we observe that our model trained solely with WEBIN-
STRUCT achieves significant improvements over the base models. For instance, MAmmoTH2-7B boosts
the performance of Mistral-7B by an average of 14 points. Notably, WEBINSTRUCT does not contain
any training data from these evaluation benchmarks, making all evaluations essentially held-out. The
substantial performance gains demonstrate the strong generalization capabilities of MAmmoTH2-7B.
Similarly, MAmmoTH2-8B boosts the performance of Llama-3-8B-base by an average of 8.8 points.
We also experiment with larger models like Yi-34B and Mixtral to show that the performance gains
are consistent across the board. Notably, Yi-34B’s performance on MATH also increases by 19%
after training on WEBINSTRUCT.
Further tuning on several additional public datasets also significantly enhances performance. The
MAmmoTH2-Plus model family achieves state-of-the-art results across the board. For example,
MAmmoTH2-Plus’s performance on TheoremQA, GPQA, and ARC-C represents the best-known
results for any model under 10B parameters. MAmmoTH2-7B-Plus’s performance on MATH and
GSM is also close to the best-known results. We also show the results of the models solely trained on
the additional public datasets in Appendix E.
An interesting comparison is between MAmmoTH2-8B-Plus and Llama3-Instruct, as both models
are trained from the Llama3-base. Llama-3-instruct was trained on a 10M human-annotated in-
struction dataset along with public datasets, similar to WEBINSTRUCT combined with additional
public datasets. Therefore, these two models are highly comparable. Our experiments show that
MAmmoTH2-8B-Plus outperforms Llama3-Instruct by an average of 6% across the benchmarks. This
substantial gain indicates that WEBINSTRUCT is highly cost-effective. For larger models, we found
that MAmmoTH2-8x7B-Plus can even match the performance of Qwen-1.5-110B with only 13B active
parameters. These results demonstrate the effectiveness of our scalable instruction tuning approach.
4.2
Additional Experimental Results
To further demonstrate the capabilities of our models beyond the reasoning benchmarks presented
in Table 2, we conduct additional experiments to evaluate their performance on code generation,
general language understanding, and instruction-following tasks. Table 3 showcases the results of var-
ious models on code generation tasks. The MAmmoTH2-7B-Plus model exhibits strong performance,
achieving the highest average scores of 66.1 and 58.2 on HumanEval(+) and MBPP(+) datasets,
respectively. It outperforms the official instruct counterparts like Mistral-7B-Instruct-v0.2 on these
metrics, indicating its superior code generation abilities.
To assess the general language understanding and instruction-following capabilities of our models, we
evaluate them on a range of benchmarks, as shown in Table 3. The MAmmoTH2-Plus models exhibit
strong performance across these tasks, showcasing their versatility and robustness. For example,
MAmmoTH2-8×7B-Plus achieves the highest scores on AlpacaEval 2.0 and Arena Hard leaderboards,
surpassing competitive models like GPT-3.5-Turbo and Tulu-2-DPO-70B (Ivison et al., 2023).
2M
4M
6M
8M 10M
# of Instructions
20
25
30
Accuracy (%)
MATH
2M
4M
6M
8M 10M
# of Instructions
15
20
25
Accuracy (%)
TheoremQA
2M
4M
6M
8M 10M
# of Instructions
72.5
75.0
77.5
80.0
Accuracy (%)
ARC-C
Extracted QA (LM Loss)
Refined QA (LM Loss)
Refined QA (SFT Loss)
Figure 5: Mistral-7B model reasoning performance improves with scaling instructions. Additionally,
SFT Loss is a more effective learning approach compared to LM Loss.
7


Table 3: Evaluation of code generation, instruction-following and MMLU(-Pro) performance for
various models. We report the average of HumanEval(+) and MBPP (+) accuracy as the code
generation performance (breakdown results are in Appendix D). Baseline scores are sourced
from the original papers or the EvalPlus, MT-Bench, AlpacaEval 2.0, Arena Hard and MMLU-Pro
leaderboards. (“-”) indicates that the score is not available from the sources. MAmmoTH2-Plus
exhibits strong general conversational ability and excels at multitask language understanding across a
wide range of domains compared to their official instruct counterparts and larger models.
Code
Generation
MT-Bench
Alpaca
Eval 2.0
Arena
Hard
MMLU
MMLU-Pro
GPT-4-1106-preview
85.6 (77.5)
9.32
50.0
-
-
-
GPT-3.5-Turbo-1106
79.7 (70.2)
8.32
19.3
18.9
-
-
GPT-3.5-Turbo-0301
-
7.94
18.1
18.1
70.0
-
Tulu-2-DPO-70B
51.2 (43.0)
7.89
21.2
15.0
67.8
40.5
Llama-2-70b-chat
31.4 (26.5)
6.86
14.7
11.6
63.0
33.6
Yi-34B-Chat
38.7 (32.6)
7.86
27.2
23.1
73.5
42.1
Mistral-7B-Instruct-v0.2
43.4 (36.5)
7.60
17.1
12.6
60.8
30.8
Llama-3-8B-Instruct
65.8 (58.0)
8.02
22.9
20.6
67.2
40.9
Mixtral-8×7B-Instruct-v0.1
52.3 (44.7)
8.30
23.7
23.4
70.6
41.0
MAmmoTH2-7B-Plus
66.1 (58.2)
7.88
23.4
14.6
63.3
40.9
MAmmoTH2-8B-Plus
61.9 (53.3)
7.95
18.5
16.6
64.6
43.4
MAmmoTH2-8x7B-Plus
63.3 (55.3)
8.20
33.8
32.6
68.3
50.4
The strong performance of MAmmoTH2 on code generation and general language understanding tasks,
as evidenced by Table 3, demonstrates that our method does not overfit to the reasoning benchmarks.
Instead, it shows the models’ ability to generalize well to a wide range of tasks, highlighting their
versatility and robustness. These additional experiments further validate the effectiveness of our
WEBINSTRUCT in developing powerful and flexible language models.
5
Ablation Study
5.1
Scaling Effect of Instructions
We first investigate the impact of model scaling and loss functions on the performance of language
models across three representative tasks: MATH, TheoremQA, and ARC-C. We train models with
varying training samples (1M to 10M) using extracted QA and refined QA data, and compare the
effectiveness of two training losses: LM loss and SFT loss. Figure 5 shows that increasing model
size and using SFT Loss with synthetic data consistently improves accuracy across all tasks. These
findings demonstrate the importance of model scaling and supervised fine-tuning with synthetic data
for enhancing language model performance in various domains.
5.2
Comparison of Two Refined Models
Table 4: Comparison of the two data-refining LLMs. We
train the three models with the same steps.
Data
GSM
MATH
MMLU-S
Theo.
ARC.
Mixtral
62.9
29.1
56.5
26.1
78.3
Qwen
65.4
28.9
60.6
23.5
80.8
Merged
65.6
31.0
60.5
24.8
81.8
To assess the effectiveness of the Q-
A pair refinement process by different
LLMs, we conducted experiments by
training three mistral-7B models: one on
the data refined by Mixtral-22B×8, an-
other on the data refined by Qwen-72B,
and a third on the merged samples re-
fined by both models. For a fair compari-
son, we trained the models with the same
9000 steps and a global batch size of 512. Our results show that the model trained on Mixtral-22B×8
refined data achieves comparable performance to the one trained on Qwen-72B refined data. The
model trained on the merged samples consistently outperforms the models trained on data refined by
individual LLMs. This demonstrates the effectiveness of using multiple LLMs for refinement, as it
leads to a more diverse and comprehensive dataset.
8


5.3
Comparison of Different Domains and Sources.
To understand how each domain (e.g., math, science, others) and data source (e.g., forum websites
and education websites) contribute to the training, we train Mistral 7B on the subsets of different
domains and data sources. Details of how we obtain domain labels can be found in Appendix G.
Table 5: Impact of different data domains and sources.
Data Source
GSM
MATH
MMLU-S
Theo.
ARC.
Base
47.4
15.7
51.4
17.3
77.6
Forum
51.0
24.0
54.7
21.0
78.2
Education
58.0
24.8
54.3
23.2
79.5
Math
52.9
27.3
51.6
21.7
74.1
Science
54.4
23.7
58.9
21.0
83.6
Other
59.4
20.8
55.3
21.1
79.4
As shown in Table 5, training on differ-
ent domains and data sources leads to
varied performance across the evaluation
benchmarks. The education website data
source consistently outperforms the fo-
rum data source, indicating the higher
quality of educational questions. Inter-
estingly, while the math domain excels
on MATH, it does not lead to significant
improvements on GSM8K, another math-
focused dataset, suggesting that training
on a single math dataset may not gener-
alize well to other math benchmarks. Furthermore, training solely on the math domain does not yield
substantial gains on science and STEM benchmarks, highlighting the need for diverse training. In
contrast, the "Other" domain, which includes a diverse range of subjects, achieves the highest score
on GSM8K, emphasizing the importance of diversity in the training data.
5.4
Case Study
Figure 6: Quality distribution of 50 sam-
pled refined QA examples.
We further conduct a case study examining the quality
of extracted and refined QA pairs from the dataset. We
showcase some good and bad cases in Appendix J. We
observe that the question/answer pairs extracted from well-
formed exam and education websites are of high quality.
The common issue is that a large portion of extracted
answers do not contain intermediate rationale (chain-of-
thought). This issue could lead to worse generalization.
Therefore, we prompt Mixtral and Qwen-72B to complete
the intermediate steps. We observe that the success rate
of such completion is relatively high. However, there are
cases where the extracted question/answer pairs contain
serious formatting issues, which pose challenges for the
following refinement step. Besides these issues, we also
observe that LLMs can sometimes modify the intention of
the originally extracted content, causing hallucinations.
To quantify the error percentages, we randomly sample 50 refined QA examples and ask the human
annotators to compare whether the refined examples are correct and significantly better than the
extracted ones in terms of format and intermediate solutions. As we can see from Figure 6, 78%
examples have been improved after refinement and only 10% examples introduce hallucinations after
refinement. Overall, our case study reveals that the harvested instruction tuning dataset is generally
accurate with a low error rate.
6
Conclusion
In this paper, we argue that the web corpus contains a vast amount of high-quality instruction data
across various domains. To mine this data, we develop a three-step pipeline consisting of recall,
extraction, and refinement steps. Through this pipeline, we harvest WEBINSTRUCT, a total of
10M diverse, high-quality instruction-response pairs and train language models. Our experiments
demonstrate that MAmmoTH2 exhibits significantly enhanced science reasoning abilities compared to
the baseline models. Our work showcases the potential of harnessing the vast amount of instruction
data in the web corpus to democratize the development of LLMs with enhanced reasoning capabilities.
9


References
J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt,
S. Altman, S. Anadkat, et al. Gpt-4 technical report. ArXiv preprint, abs/2303.08774, 2023. URL
https://arxiv.org/abs/2303.08774.
A. Amini, S. Gabriel, S. Lin, R. Koncel-Kedziorski, Y. Choi, and H. Hajishirzi. MathQA: Towards
interpretable math word problem solving with operation-based formalisms. In Proceedings of the
2019 Conference of the North American Chapter of the Association for Computational Linguistics:
Human Language Technologies, Volume 1 (Long and Short Papers), pages 2357–2367, 2019. doi:
10.18653/v1/N19-1245. URL https://aclanthology.org/N19-1245.
J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry,
Q. Le, et al. Program synthesis with large language models. ArXiv preprint, abs/2108.07732, 2021.
URL https://arxiv.org/abs/2108.07732.
Z. Azerbayev, H. Schoelkopf, K. Paster, M. Dos Santos, S. M. McAleer, A. Q. Jiang, J. Deng,
S. Biderman, and S. Welleck. Llemma: An open language model for mathematics. In The Twelfth
International Conference on Learning Representations, 2023. URL https://openreview.net/
forum?id=4WnqRR915j.
J. Bai, S. Bai, Y. Chu, Z. Cui, K. Dang, X. Deng, Y. Fan, W. Ge, Y. Han, F. Huang, et al. Qwen
technical report. ArXiv preprint, abs/2309.16609, 2023. URL https://arxiv.org/abs/2309.
16609.
L. Ben Allal, A. Lozhkov, G. Penedo, T. Wolf, and L. von Werra. Cosmopedia, 2024. URL
https://huggingface.co/datasets/HuggingFaceTB/cosmopedia.
T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam,
G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh,
D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark,
C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Language models are few-shot
learners. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in
Neural Information Processing Systems 33: Annual Conference on Neural Information Processing
Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL https://proceedings.
neurips.cc/paper/2020/hash/1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html.
M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y. Burda,
N. Joseph, G. Brockman, et al.
Evaluating large language models trained on code.
ArXiv
preprint, abs/2107.03374, 2021. URL https://arxiv.org/abs/2107.03374.
W. Chen, X. Ma, X. Wang, and W. W. Cohen. Program of thoughts prompting: Disentangling
computation from reasoning for numerical reasoning tasks. Transactions on Machine Learning
Research, 2023a. URL https://openreview.net/forum?id=YfZ4ZPt8zd.
W. Chen, M. Yin, M. Ku, P. Lu, Y. Wan, X. Ma, J. Xu, X. Wang, and T. Xia. Theoremqa: A theorem-
driven question answering dataset. In The 2023 Conference on Empirical Methods in Natural
Language Processing, 2023b. URL https://aclanthology.org/2023.emnlp-main.489/.
H. W. Chung, L. Hou, S. Longpre, B. Zoph, Y. Tay, W. Fedus, Y. Li, X. Wang, M. Dehghani,
S. Brahma, et al. Scaling instruction-finetuned language models. Journal of Machine Learning
Research, 25(70):1–53, 2024. URL https://arxiv.org/pdf/2210.11416.
P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord. Think you have
solved question answering? try arc, the ai2 reasoning challenge. ArXiv preprint, abs/1803.05457,
2018. URL https://arxiv.org/abs/1803.05457.
K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton,
R. Nakano, et al. Training verifiers to solve math word problems. ArXiv preprint, abs/2110.14168,
2021. URL https://arxiv.org/abs/2110.14168.
T. Computer. Redpajama: an open dataset for training large language models, 2023. URL https:
//github.com/togethercomputer/RedPajama-Data.
10


L. Gao, A. Madaan, S. Zhou, U. Alon, P. Liu, Y. Yang, J. Callan, and G. Neubig. Pal: Program-aided
language models. In International Conference on Machine Learning, pages 10764–10799. PMLR,
2023. URL https://arxiv.org/pdf/2211.10435.
Z. Gou, Z. Shao, Y. Gong, Y. Yang, M. Huang, N. Duan, W. Chen, et al. Tora: A tool-integrated
reasoning agent for mathematical problem solving. ArXiv preprint, abs/2309.17452, 2023. URL
https://arxiv.org/abs/2309.17452.
D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Measur-
ing massive multitask language understanding. In 9th International Conference on Learning
Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021, 2021a.
URL https:
//openreview.net/forum?id=d7KBjmI3GmQ.
D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt.
Measuring mathematical problem solving with the math dataset. In Thirty-fifth Conference on
Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021b. URL
https://openreview.net/forum?id=7Bywt2mQsCe.
H. Ivison, Y. Wang, V. Pyatkin, N. Lambert, M. Peters, P. Dasigi, J. Jang, D. Wadden, N. A. Smith,
I. Beltagy, et al. Camels in a changing climate: Enhancing lm adaptation with tulu 2. ArXiv
preprint, abs/2311.10702, 2023. URL https://arxiv.org/pdf/2311.10702.
A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. d. l. Casas, F. Bressand,
G. Lengyel, G. Lample, L. Saulnier, et al. Mistral 7b. ArXiv preprint, abs/2310.06825, 2023. URL
https://arxiv.org/abs/2310.06825.
A. Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bamford, D. S. Chaplot, D. d. l.
Casas, E. B. Hanna, F. Bressand, et al. Mixtral of experts. ArXiv preprint, abs/2401.04088, 2024.
URL https://arxiv.org/abs/2401.04088.
A. Joulin, E. Grave, P. Bojanowski, M. Douze, H. Jégou, and T. Mikolov. Fasttext. zip: Compressing
text classification models. ArXiv preprint, abs/1612.03651, 2016. URL https://arxiv.org/
abs/1612.03651.
A. Lewkowycz, A. Andreassen, D. Dohan, E. Dyer, H. Michalewski, V. Ramasesh, A. Slone,
C. Anil, I. Schlag, T. Gutman-Solo, et al. Solving quantitative reasoning problems with language
models. Advances in Neural Information Processing Systems, 35:3843–3857, 2022. URL https:
//openreview.net/forum?id=IFXTZERXdM7.
C. Li, W. Wang, J. Hu, Y. Wei, N. Zheng, H. Hu, Z. Zhang, and H. Peng. Common 7b language
models already possess strong math capabilities. ArXiv preprint, abs/2403.04706, 2024a. URL
https://arxiv.org/abs/2403.04706.
T. Li, W.-L. Chiang, L. D. Evan Frick, B. Zhu, J. E. Gonzalez, and I. Stoica. From live data to
high-quality benchmarks: The arena-hard pipeline, April 2024b. URL https://lmsys.org/
blog/2024-04-19-arena-hard/.
X. Li, T. Zhang, Y. Dubois, R. Taori, I. Gulrajani, C. Guestrin, P. Liang, and T. B. Hashimoto.
Alpacaeval: An automatic evaluator of instruction-following models. https://github.com/
tatsu-lab/alpaca_eval, 2023.
Z. Lin, Z. Gou, T. Liang, R. Luo, H. Liu, and Y. Yang. Criticbench: Benchmarking llms for critique-
correct reasoning. ArXiv preprint, abs/2402.14809, 2024. URL https://arxiv.org/abs/2402.
14809.
J. Liu, C. S. Xia, Y. Wang, and L. Zhang.
Is your code generated by chatgpt really correct?
rigorous evaluation of large language models for code generation. Advances in Neural Information
Processing Systems, 36, 2024. URL https://openreview.net/pdf?id=1qvx610Cu7.
H. Luo, Q. Sun, C. Xu, P. Zhao, J. Lou, C. Tao, X. Geng, Q. Lin, S. Chen, and D. Zhang. Wizardmath:
Empowering mathematical reasoning for large language models via reinforced evol-instruct. ArXiv
preprint, abs/2308.09583, 2023. URL https://arxiv.org/abs/2308.09583.
11


Meta. Introducing meta llama 3: The most capable openly available llm to date. https://ai.meta.
com/blog/meta-llama-3/, April 2024.
A. Mitra, H. Khanpour, C. Rosset, and A. Awadallah. Orca-math: Unlocking the potential of slms
in grade school math. ArXiv preprint, abs/2402.14830, 2024. URL https://arxiv.org/abs/
2402.14830.
M. Nye, A. J. Andreassen, G. Gur-Ari, H. Michalewski, J. Austin, D. Bieber, D. Dohan,
A. Lewkowycz, M. Bosma, D. Luan, et al.
Show your work: Scratchpads for intermedi-
ate computation with language models. In Deep Learning for Code Workshop, 2022. URL
https://openreview.net/forum?id=iedYJm92o0a.
L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agar-
wal,
K. Slama,
A. Ray,
et al.
Training language models to follow instructions
with human feedback.
Advances in neural information processing systems, 35:27730–
27744, 2022. URL https://proceedings.neurips.cc/paper_files/paper/2022/file/
b1efde53be364a73914f58805a001731-Paper-Conference.pdf.
K. Paster, M. Dos Santos, Z. Azerbayev, and J. Ba. Openwebmath: An open dataset of high-quality
mathematical web text. In The Twelfth International Conference on Learning Representations,
2023. URL https://openreview.net/pdf?id=jKHmjlpViu.
B. Peng, C. Li, P. He, M. Galley, and J. Gao. Instruction tuning with gpt-4. ArXiv preprint,
abs/2304.03277, 2023. URL https://arxiv.org/abs/2304.03277.
J. Rasley, S. Rajbhandari, O. Ruwase, and Y. He. Deepspeed: System optimizations enable training
deep learning models with over 100 billion parameters. In R. Gupta, Y. Liu, J. Tang, and B. A.
Prakash, editors, KDD ’20: The 26th ACM SIGKDD Conference on Knowledge Discovery and
Data Mining, Virtual Event, CA, USA, August 23-27, 2020, pages 3505–3506, 2020.
URL
https://dl.acm.org/doi/10.1145/3394486.3406703.
D. Rein, B. L. Hou, A. C. Stickland, J. Petty, R. Y. Pang, J. Dirani, J. Michael, and S. R. Bowman.
Gpqa: A graduate-level google-proof q&a benchmark. ArXiv preprint, abs/2311.12022, 2023.
URL https://arxiv.org/abs/2311.12022.
V. Sanh, A. Webson, C. Raffel, S. H. Bach, L. Sutawika, Z. Alyafeai, A. Chaffin, A. Stiegler, A. Raja,
M. Dey, M. S. Bari, C. Xu, U. Thakker, S. S. Sharma, E. Szczechla, T. Kim, G. Chhablani, N. V.
Nayak, D. Datta, J. Chang, M. T. Jiang, H. Wang, M. Manica, S. Shen, Z. X. Yong, H. Pandey,
R. Bawden, T. Wang, T. Neeraj, J. Rozen, A. Sharma, A. Santilli, T. Févry, J. A. Fries, R. Teehan,
T. L. Scao, S. Biderman, L. Gao, T. Wolf, and A. M. Rush. Multitask prompted training enables
zero-shot task generalization. In The Tenth International Conference on Learning Representations,
ICLR 2022, Virtual Event, April 25-29, 2022, 2022. URL https://openreview.net/forum?
id=9Vrb9D0WI4.
Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, M. Zhang, Y. Li, Y. Wu, and D. Guo. Deepseekmath: Push-
ing the limits of mathematical reasoning in open language models. ArXiv preprint, abs/2402.03300,
2024. URL https://arxiv.org/abs/2402.03300.
A. Srivastava, A. Rastogi, A. Rao, A. A. M. Shoeb, A. Abid, A. Fisch, A. R. Brown, A. Santoro,
A. Gupta, A. Garriga-Alonso, et al. Beyond the imitation game: Quantifying and extrapolating
the capabilities of language models. Transactions on Machine Learning Research, 2023. URL
https://openreview.net/forum?id=uyTL5Bvosj.
L. Sun, Y. Han, Z. Zhao, D. Ma, Z. Shen, B. Chen, L. Chen, and K. Yu. Scieval: A multi-level large
language model evaluation benchmark for scientific research. ArXiv preprint, abs/2308.13149,
2023. URL https://arxiv.org/abs/2308.13149.
M. Suzgun, N. Scales, N. Schärli, S. Gehrmann, Y. Tay, H. W. Chung, A. Chowdhery, Q. V. Le, E. H.
Chi, D. Zhou, , and J. Wei. Challenging big-bench tasks and whether chain-of-thought can solve
them. ArXiv preprint, abs/2210.09261, 2022. URL https://arxiv.org/abs/2210.09261.
Z. Tang, X. Zhang, B. Wan, and F. Wei. Mathscale: Scaling instruction tuning for mathematical
reasoning. ArXiv preprint, abs/2403.02884, 2024. URL https://arxiv.org/abs/2403.02884.
12


R. Taori, I. Gulrajani, T. Zhang, Y. Dubois, X. Li, C. Guestrin, P. Liang, and T. B. Hashimoto.
Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/
stanford_alpaca, 2023.
R. Taylor, M. Kardas, G. Cucurull, T. Scialom, A. Hartshorn, E. Saravia, A. Poulton, V. Kerkez, and
R. Stojnic. Galactica: A large language model for science. ArXiv preprint, abs/2211.09085, 2022.
URL https://arxiv.org/abs/2211.09085.
G. Team, R. Anil, S. Borgeaud, Y. Wu, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M.
Dai, A. Hauth, et al. Gemini: a family of highly capable multimodal models. ArXiv preprint,
abs/2312.11805, 2023. URL https://arxiv.org/abs/2312.11805.
Teknium. Openhermes 2.5: An open dataset of synthetic data for generalist llm assistants, 2023.
URL https://huggingface.co/datasets/teknium/OpenHermes-2.5.
TIGER-Lab. Mmlu professional dataset. Hugging Face Dataset Hub, 2024. URL https://
huggingface.co/datasets/TIGER-Lab/MMLU-Pro.
S. Toshniwal, I. Moshkov, S. Narenthiran, D. Gitman, F. Jia, and I. Gitman. Openmathinstruct-1:
A 1.8 million math instruction tuning dataset. ArXiv preprint, abs/2402.10176, 2024. URL
https://arxiv.org/abs/2402.10176.
H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal,
E. Hambro, F. Azhar, et al. Llama: Open and efficient foundation language models. ArXiv preprint,
abs/2302.13971, 2023a. URL https://arxiv.org/abs/2302.13971.
H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra,
P. Bhargava, S. Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. ArXiv
preprint, abs/2307.09288, 2023b. URL https://arxiv.org/abs/2307.09288.
X. Wang, Z. Hu, P. Lu, Y. Zhu, J. Zhang, S. Subramaniam, A. Loomba, S. Zhang, Y. Sun, and
W. Wang. Scibench: Evaluating college-level scientific problem-solving abilities of large language
models. In The 3rd Workshop on Mathematical Reasoning and AI at NeurIPS’23, 2023a. URL
https://openreview.net/forum?id=u6jbcaCHqO.
Y. Wang, S. Mishra, P. Alipoormolabashi, Y. Kordi, A. Mirzaei, A. Naik, A. Ashok, A. S.
Dhanasekaran, A. Arunkumar, D. Stap, E. Pathak, G. Karamanolakis, H. Lai, I. Purohit, I. Mon-
dal, J. Anderson, K. Kuznia, K. Doshi, K. K. Pal, M. Patel, M. Moradshahi, M. Parmar,
M. Purohit, N. Varshney, P. R. Kaza, P. Verma, R. S. Puri, R. Karia, S. Doshi, S. K. Sampat,
S. Mishra, S. Reddy A, S. Patro, T. Dixit, and X. Shen. Super-NaturalInstructions: General-
ization via declarative instructions on 1600+ NLP tasks. In Proceedings of the 2022 Confer-
ence on Empirical Methods in Natural Language Processing, pages 5085–5109, 2022. URL
https://aclanthology.org/2022.emnlp-main.340.
Y. Wang, Y. Kordi, S. Mishra, A. Liu, N. A. Smith, D. Khashabi, and H. Hajishirzi. Self-instruct:
Aligning language models with self-generated instructions. In Proceedings of the 61st Annual
Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13484–
13508, 2023b. URL https://arxiv.org/abs/2212.10560.
Z. Wang, R. Xia, and P. Liu. Generative ai for math: Part i–mathpile: A billion-token-scale pretraining
corpus for math. ArXiv preprint, abs/2312.17120, 2023c. URL https://arxiv.org/abs/2312.
17120.
J. Wei, M. Bosma, V. Y. Zhao, K. Guu, A. W. Yu, B. Lester, N. Du, A. M. Dai, and Q. V. Le.
Finetuned language models are zero-shot learners. In The Tenth International Conference on
Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022, 2022a. URL https:
//openreview.net/forum?id=gEZrGCozdqR.
J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhou, et al. Chain-of-thought
prompting elicits reasoning in large language models. Advances in neural information processing
systems, 35:24824–24837, 2022b. URL https://openreview.net/pdf?id=_VjQlMeSB_J.
13


C. Xu, Q. Sun, K. Zheng, X. Geng, P. Zhao, J. Feng, C. Tao, and D. Jiang. Wizardlm: Empowering
large language models to follow complex instructions. ArXiv preprint, abs/2304.12244, 2023. URL
https://arxiv.org/abs/2304.12244.
H. Ying, S. Zhang, L. Li, Z. Zhou, Y. Shao, Z. Fei, Y. Ma, J. Hong, K. Liu, Z. Wang, et al.
Internlm-math: Open math large language models toward verifiable reasoning. ArXiv preprint,
abs/2402.06332, 2024. URL https://arxiv.org/abs/2402.06332.
A. Young, B. Chen, C. Li, C. Huang, G. Zhang, G. Zhang, H. Li, J. Zhu, J. Chen, J. Chang,
et al. Yi: Open foundation models by 01. ai. ArXiv preprint, abs/2403.04652, 2024. URL
https://arxiv.org/abs/2403.04652.
L. Yu, W. Jiang, H. Shi, Y. Jincheng, Z. Liu, Y. Zhang, J. Kwok, Z. Li, A. Weller, and W. Liu.
Metamath: Bootstrap your own mathematical questions for large language models. In The Twelfth
International Conference on Learning Representations, 2023. URL https://openreview.net/
forum?id=N8N0hgNDRt.
L. Yuan, G. Cui, H. Wang, N. Ding, X. Wang, J. Deng, B. Shan, H. Chen, R. Xie, Y. Lin, et al.
Advancing llm reasoning generalists with preference trees. ArXiv preprint, abs/2404.02078, 2024.
URL https://arxiv.org/abs/2404.02078.
X. Yue, Y. Ni, K. Zhang, T. Zheng, R. Liu, G. Zhang, S. Stevens, D. Jiang, W. Ren, Y. Sun, et al.
Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert
agi. ArXiv preprint, abs/2311.16502, 2023a. URL https://arxiv.org/abs/2311.16502.
X. Yue, X. Qu, G. Zhang, Y. Fu, W. Huang, H. Sun, Y. Su, and W. Chen. Mammoth: Building math
generalist models through hybrid instruction tuning. In The Twelfth International Conference on
Learning Representations, 2023b. URL https://openreview.net/forum?id=yLClGs770I.
Y. Zhang, Y. Luo, Y. Yuan, and A. C.-C. Yao. Automathtext: Autonomous data selection with
language models for mathematical texts. ArXiv preprint, abs/2402.07625, 2024. URL https:
//arxiv.org/abs/2402.07625.
L. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. Xing,
et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information
Processing Systems, 36, 2024a. URL https://openreview.net/forum?id=uccHPGDlao.
T. Zheng, S. Guo, X. Qu, J. Guo, W. Zhang, X. Du, C. Lin, W. Huang, W. Chen, J. Fu, et al. Kun:
Answer polishment for chinese self-alignment with instruction back-translation. ArXiv preprint,
abs/2401.06477, 2024b. URL https://arxiv.org/abs/2401.06477.
T. Zheng, G. Zhang, T. Shen, X. Liu, B. Y. Lin, J. Fu, W. Chen, and X. Yue. Opencodeinterpreter:
Integrating code generation with execution and refinement. ArXiv preprint, abs/2402.14658, 2024c.
URL https://arxiv.org/abs/2402.14658.
Y. Zheng, R. Zhang, J. Zhang, Y. Ye, Z. Luo, and Y. Ma. Llamafactory: Unified efficient fine-tuning
of 100+ language models. ArXiv preprint, abs/2403.13372, 2024d. URL https://arxiv.org/
abs/2403.13372.
14


A
Details of Additional Public Instruction Tuning Datasets
• OpenHermes 2.5 (Teknium, 2023): The OpenHermes-2.5 dataset is a comprehensive collection
of diverse data sources for instruction tuning, including 1M examples from datasets in math,
science, and coding, alongside synthetic and chat-based data. It incorporates diverse sources such
as Airoboros 2.2, CamelAI Domain Expert Datasets, ChatBot Arena, Collective Cognition, Evol
Instruct, Glaive Code Assistant, GPT4-LLM, GPTeacher, Medical Tasks, MetaMath, SlimOrca,
Platypus, ShareGPT, and Unnatural Instructions GPT4. We remove TheoremQA from Platypus as
it is one of our test sets.
• Code-Feedback (Zheng et al., 2024c): The Code-Feedback dataset is a multi-turn code generation
and refinement dataset, containing 68,000 multi-turn interactions between users, code generation
models, and compiler systems. It includes initial user instructions followed by compiler and user
feedback. This dataset significantly enhances the model’s multi-turn interaction coding ability.
• Math-Plus: This dataset combines public datasets such as MetaMathQA (395K) (Yu et al., 2023)
and Orca-Math (200K) (Mitra et al., 2024). Both of these datasets are generated by GPT-3.5/4
using GSM/MATH and other math datasets as the seed data. To further augment the dataset, we
prompt GPT-4 to rewrite Q-A pairs from MATH training sets, adding an additional 300K examples
to enhance the challenging problems. The total size of the Math-Plus dataset is 894K examples.
To ensure consistency and compatibility, we carefully align the format and structure of these additional
datasets with WEBINSTRUCT. These supplementary datasets provide a rich resource for training
models to answer questions and provide explanations across a wide range of topics, enhancing their
versatility and applicability in real-world scenarios.
B
Details of Evaluation Datasets
To rigorously assess the capabilities of models in reasoning abilities across different domains, we
utilize several widely used datasets. Each of these datasets is designed to challenge the models in
different aspects of reasoning.
• GSM8K (Cobbe et al., 2021): This test dataset contains 1.32K diverse grade school math problems,
intended to test basic arithmetic and reasoning ability in an educational context.
• MATH (Hendrycks et al., 2021b): Comprising 5000 intricate competition-level problems to
evaluate the models’ ability to perform complex mathematical reasoning.
• TheoremQA (Chen et al., 2023b): Focused on applying mathematical theorems to solve advanced
problems in fields such as mathematics, physics, and engineering, TheoremQA includes 800
questions that test the theoretical reasoning capabilities.
• BIG-Bench Hard (BBH) (Suzgun et al., 2022): Consisting of 23 tasks previously found challeng-
ing for language models from BIG-Bench (Srivastava et al., 2023), BBH contains a total of 6511
challenging problems examining the capability of LLMs to solve them.
• ARC-C (Clark et al., 2018): ARC includes questions derived from various grade-level science
exams, testing models’ ability to handle both straightforward and complex scientific queries. We
use the challenge subset, which contains 1,172 test questions.
• GPQA (Rein et al., 2023): This dataset provides "Google-proof" questions in biology, physics, and
chemistry, designed to test deep domain expertise and reasoning under challenging conditions. We
use the diamond subset containing 198 hard problems.
• MMLU-STEM (Hendrycks et al., 2021a): Spanning 57 subjects across multiple disciplines,
MMLU evaluates the breadth and depth of a model’s knowledge in a manner akin to academic and
professional testing environments. We select the STEM subset of MMLU with 3.13K problems.
These datasets collectively enable a comprehensive assessment of language models’ reasoning
prowess across a spectrum of complexity and realism. We further evaluate the models on additional
code generation tasks (including HumanEval (Chen et al., 2021), MBPP (Austin et al., 2021) and
their augmented version (Liu et al., 2024)), general LLM benchmarks like MMLU (Hendrycks
et al., 2021a), and chat benchmarks like MT-Bench (Zheng et al., 2024a), AlpacaEval 2.0 (Li et al.,
2023), and Arena Hard (Li et al., 2024b) to demonstrate the generalizability of WEBINSTRUCT and
WEBINSTRUCT-PLUS on more general LLM benchmarks.
15


C
Related Work
Instruction Tuning.
Instruction tuning is crucial for aligning large language models (LLMs) with
end tasks. There are two main types of instruction tuning data: (1) human-written data, such as
FLAN (Wei et al., 2022a), T0 (Sanh et al., 2022), and SuperNI (Wang et al., 2022), which assemble
large instruction-tuning datasets from existing human-labeled datasets; and (2) synthesized data, like
Self-Instruct (Wang et al., 2023b), WizardLM (Xu et al., 2023), and GPT4-Alpaca (Peng et al., 2023),
which create instruction-tuning datasets by synthesizing from powerful LLMs like GPT-4 (Achiam
et al., 2023). Both types of instruction-tuning data have advantages and disadvantages. Human-
written data is limited in size due to the high cost and in task diversity because existing human-labeled
datasets mostly focus on a few NLP tasks. Although synthesized data can be generated at any scale,
the high rate of hallucination can lead to significant quality degradation. Moreover, the diversity of
synthesized data is heavily influenced by the seed data. Without a diverse seed dataset, the synthesized
data will lack domain coverage.
Mathematics Reasoning.
In recent years, there has been a growing interest in enhancing the
mathematical reasoning abilities of large language models (LLMs). Three main approaches have
been proposed to improve LLMs’ mathematical reasoning skills:
• Prompting: Chain-of-thought-prompting (CoT) (Nye et al., 2022; Wei et al., 2022b) elicits LLMs’
inherent reasoning ability by demonstrating intermediate reasoning steps. Program-of-thoughts-
prompting (PoT) (Chen et al., 2023a; Gao et al., 2023) utilizes tools to further augment LLMs’
math reasoning abilities. Subsequent work (Gou et al., 2023; Toshniwal et al., 2024; Yuan et al.,
2024) combines CoT and PoT to maximize LLMs’ reasoning ability.
• Continued Training: Enabling LLMs to solve mathematical problems has been a long-standing
challenge. MINERVA (Lewkowycz et al., 2022) and Galactica (Taylor et al., 2022) were pioneers
in continued training of LLMs to adapt to scientific domains for math and science reasoning.
Open-source models like Llemma (Azerbayev et al., 2023), DeepSeek-Math (Shao et al., 2024),
and Intern-Math (Ying et al., 2024) have surpassed MINERVA and Galactica on math benchmarks.
These approaches mainly rely on using an efficient classifier to recall documents from Common
Crawl to retrieve a massive high-quality math-related corpus (>100B tokens) to enhance math
reasoning.
• Instruction Tuning: Instruction tuning aims to enhance LLMs’ math reasoning skills by efficiently
training on human-annotated public datasets like GSM8K (Cobbe et al., 2021), MATH (Hendrycks
et al., 2021b), and MathQA (Amini et al., 2019). However, these datasets are often insufficient
in size and diversity. Therefore, recent work (Yu et al., 2023; Yue et al., 2023b; Toshniwal et al.,
2024; Luo et al., 2023; Li et al., 2024a) proposes augmenting them with strong commercial LLMs
like GPT-4 (Achiam et al., 2023). These methods can significantly boost LLMs’ performance on
in-domain math benchmarks but may fall short of generalization.
Our work combines continued training with instruction tuning to exploit the benefits of both ap-
proaches. Specifically, our dataset is recalled from Common Crawl like DeepSeekMath. However,
due to the significant level of noise in the raw corpus, we utilize a strong LLM to filter and clean the
corpus to extract the instruction tuning pairs.
Science Reasoning.
In addition to mathematical reasoning, there is growing interest in improving
LLMs’ general scientific reasoning ability in subjects like physics, biology, chemistry, computer
science, etc. Several benchmarks, such as MMLU (Hendrycks et al., 2021a), TheoremQA (Chen et al.,
2023b), Sci-Bench (Wang et al., 2023a), SciEval (Sun et al., 2023), and GPQA (Rein et al., 2023),
have been developed to measure LLMs’ reasoning ability on tasks beyond math. However, there
has been less effort in curating high-quality training data for the science domain. Most datasets, like
OpenWebMath (Paster et al., 2023), Proof-Pile (Azerbayev et al., 2023), and MathPile (Zhang et al.,
2024), are heavily biased towards mathematics. In this work, we aim to generalize the pre-training
data to broader subjects through our newly curated science seed data.
16


D
Code Generation Results
We report the code generation results of our models and baselines in Table 6.
HumanEval
HumanEval+
MBPP
MBPP+
Average
Average+
Mistral-7B
28.7
23.8
51.9
42.1
40.3
33.0
Gemma-7B
26.8
20.1
52.6
43.4
39.7
31.8
Llama-3-8B
33.5
29.3
61.4
51.6
47.5
40.5
Gemma-1.1-7B-Instruct
42.7
35.4
57.1
45.0
49.9
40.2
Mistral-7B-Instruct-v0.2
75.0
70.1
44.7
37.0
59.9
53.6
Llama-3-8B-Instruct
61.6
56.7
70.1
59.3
65.9
58.0
Mixtral-8×7B-Instruct-v0.1
45.1
39.6
59.5
49.7
52.3
44.7
MAmmoTH2– 7B-Plus
72.1
65.9
60.1
50.4
66.1
58.2
MAmmoTH2– 8B-Plus
63.4
57.9
60.4
48.6
61.9
53.3
MAmmoTH2– 8×7B-Plus
57.9
53.7
68.7
56.9
63.3
55.3
Table 6: Code generation results of different models.
Baseline results are copied from the
EvalPlus (Liu et al., 2024) leaderboard.
E
Impact of Additional Public Instruction Tuning Datasets
In Appendix A, we introduce additional public instruction tuning datasets to further boost the model’s
reasoning performance. Here, we show the three setups of models trained on: 1) WEBINSTRUCT
only; 2) Additional Public Datasets only; 3) WEBINSTRUCT + Additional Public Datasets (which
we first trained on WEBINSTRUCT and then continued training on additional public datasets). The
results are shown in Table 7.
Data
TheoremQA
MATH
GSM8K
GPQA
MMLU-ST
BBH
ARC-C
AVG
Mistral 7B Base
WEBINSTRUCT
29.0
36.7
68.4
32.4
62.4
58.6
81.7
52.8
PUBLIC DATASETS
22.6
37.9
83.5
29.3
57.6
62.7
79.9
53.4
WEBINS.+PUBLIC.
29.2
45.0
84.7
36.8
64.5
63.1
83.0
58.0
Mixtral 8x7B Base
WEBINSTRUCT
32.2
39.0
75.4
36.8
67.4
71.1
87.5
58.9
PUBLIC DATASETS
31.3
45.1
85.3
37.4
69.4
73.2
88.1
61.4
WEBINS.+PUBLIC.
34.1
47.0
86.4
37.8
72.4
74.1
88.4
62.9
Table 7: Impact of additional public instruction tuning datasets.
F
Distributions of Website Domains in WEBINSTRUCT
Figure 7 show the distribution of the top websites in WEBINSTRUCT.
17


Figure 7: The distribution of the top websites in our instruction dataset.
G
Domain Distribution of WEBINSTRUCT
Figure 8 presents a breakdown of the WEBINSTRUCT by subject domains and data sources, providing
insights into the composition and diversity of the mined instruction-response pairs. The subject labels
are automatically annotated using the Llama-3-8B-Instruct model, while the distribution between
education and forum data is obtained by analyzing the source URLs of the samples. The pie chart
reveals that WEBINSTRUCT is predominantly composed of science-related subjects, with 81.69%
of the data falling under the broad "Science" category. Within this category, Mathematics takes
up the largest share at 68.36%, followed by Physics, Chemistry, and Biology. This highlights the
dataset’s strong emphasis on mathematical problem-solving and scientific reasoning. The remaining
non-science categories, such as Business, Art & Design, and Health & Medicine, contribute to the
diversity of the dataset. In terms of data sources, the vast majority (86.73%) of the instruction-
response pairs come from exam-style questions, while forum discussions make up the remaining
13.27%. This source breakdown indicates that WEBINSTRUCT primarily consists of well-structured,
educational content, supplemented by real-world discussions and inquiries from forums. The diverse
subject coverage and the combination of education and forum data enable WEBINSTRUCT to capture
a wide range of reasoning tasks and problem-solving scenarios.
Figure 8: Breakdown of WEBINSTRUCT by subject domains and data sources.
18


H
Limitations of WEBINSTRUCT
Despite employing a three-step pipeline to ensure the quality of the mined instruction-response pairs,
there may still be some noise and inaccuracies in the dataset, as mentioned in Figure 6. The extraction
and refinement steps rely on the performance of the LLMs used, which may introduce biases and
errors. Future work could explore more advanced techniques for data cleaning and validation, such
as human-in-the-loop approaches or training a data selection model for filtering. Furthermore, while
WEBINSTRUCT covers a wide range of subjects, including math, science, and engineering, there
may be specific subdomains or advanced topics that are underrepresented, such as humanities and
other daily chat topics. Expanding the coverage of the seed data to include broader and more diverse
instruction-response pairs could further enhance the reasoning capabilities of LLMs in different areas.
I
Broader Social Impact
The development of MAmmoTH2 and the WEBINSTRUCT has the potential to positively impact edu-
cation by providing the community with access to a large-scale, diverse set of instruction-response
pairs across various subjects, particularly in mathematics and science. MAmmoTH2 can assist students
in their learning process by offering step-by-step solutions, explanations, and targeted feedback,
thus enhancing their understanding of complex concepts and problem-solving skills. Furthermore,
the dataset and models can serve as valuable resources for educators to create more effective and
engaging educational content, assignments, and assessments. However, it is important to consider
potential limitations and biases in the dataset and the model. The quality and accuracy of the
instruction-response pairs in WEBINSTRUCT may vary, and the model’s outputs may reflect these
inconsistencies. Therefore, it is crucial to employ rigorous data filtering and quality control measures
to ensure the reliability and trustworthiness of the dataset and the model.
19


J
Case Study
In this section, we aim to review some of the instances in WEBINSTRUCT. The data extracted from
the website contains many issues. For example, the most common issue is that the answer is relatively
short without any intermediate rationale, which could occur frequently from the quiz website with
multi-choice questions. Another common issue is that the extraction destroys important formatting in
the document, which leads to unrecoverable errors.
Good Case 1
Extracted:
Question: Find the distance from the point 3
2, 3
2 to the line −37x
10 −21y
10 −7
2 = 0.
Answer: 61
q
2
905.
Refined:
Question:
Calculate the distance from the point ( 3
2, 3
2) to the line defined by the equation
−37
10x −21
10y −7
2 = 0.
Answer: To find the distance ’d’ from a point (x0, y0) to a line given by Ax + By + C = 0, we use
the formula:
d = |Ax0 + By0 + C|
√
A2 + B2
.
For the point
 3
2, 3
2


and the line −37
10x −21
10y −7
2 = 0, substitute the coordinates into the formula:
d = | −37
10 · 3
2 −21
10 · 3
2 −7
2|
q−37
10

2 +
−21
10

2 .
Simplifying further, the exact simplified distance ’d’ is 61
q
2
905. Thus, the correct distance from the
point to the line is 61
q
2
905.
Figure A1: In this Refined version, the expression of both the question and the answer is clearer and
more precise. It employs standardized mathematical symbols and terms, such as representing the
coordinates of the point and the equation of the line as (x0, y0) and Ax + By + C = 0, respectively.
Moreover, it explicitly provides the formula for calculating the distance. Through these revisions,
readers can more easily understand how to compute the distance from a point to a line, making the
solution process more transparent and easier to follow.
Good Case 2
Extracted:
Question: What are the two main ways that minerals are classified?
Answer: They can be classified by luster, streak, breakage, and hardness.
Refined:
Question: What are the primary criteria used for classifying minerals in mineralogy?
Answer: Minerals are primarily classified based on two main categories: physical properties and
chemical properties. The physical properties include luster, streak, crystal form, hardness, density, and
cleavage or breakage. Chemical properties involve the mineral’s chemical composition and reactions with
other substances. These characteristics help identify and distinguish different minerals in mineralogy.
Figure A2: Refining the extracted response provides a more comprehensive and accurate explanation
of mineral classification in mineralogy. By expanding on the initial response to include both physical
and chemical properties, the refined version offers a deeper understanding of how minerals are
categorized and identified. This enriched content can enhance the model’s ability to generate
informative responses related to mineralogy.
20


Good Case 3
Extracted:
Question: How do you find theasymptotes for Q(x) = (2x2)/(x2 −5x −6)?
Answer: vertical asymptotes x=-1, x=6/n horizontal asymptote y=2 /n /n /n /n /n /n /n /n /n /n
Refined:
Question: How do you determine the vertical and horizontal asymptotes for the function Q(x) =
2x2
x2−5x−6?
Answer: To find the vertical asymptotes, we need to identify the values of x that make the denominator
equal to zero:
x2 −5x −6 = (x −6)(x + 1) = 0
This gives us the vertical asymptotes at x = −1 and x = 6.
For the horizontal asymptote, as x approaches infinity or negative infinity, the dominant terms in the
numerator and denominator are both x2. Thus, we compare their coefficients:
- Numerator’s highest degree coefficient: 2 - Denominator’s highest degree coefficient: 1
The horizontal asymptote is given by the ratio of these coefficients:
y = 2
1 = 2
Therefore, the vertical asymptotes are at x = −1 and x = 6, and the horizontal asymptote is y = 2.
Figure A3: The refined answer provides a more detailed explanation for finding the asymptotes. It
includes the steps for solving for the vertical asymptotes by factoring the denominator and determining
where it equals zero. It also explains how to find the horizontal asymptote by comparing the leading
coefficients of the highest-degree terms in both the numerator and the denominator. The use of LaTeX
formatting makes the mathematical notation clearer and easier to understand.
Good Case 4
Extracted:
Question:Which of the following is not an example of biased language? a. “You are too short to reach
that.” b. “Are you deaf?” c. “Move along, grandpa!” d. “I think I’ve been gypped.
Answer: D. I think I’ve been gypped.
Refined:
Question: Which of the following statements does not exhibit biased language?
a. “You are too short to reach that.”
b. “Are you deaf?”
c. “Move along, grandpa!”
d. “I think I’ve been gypped.”
Answer: The statement that does not exhibit biased language is:
d. "I think I’ve been gypped."
Explanation: Biased language refers to words or phrases that stereotype, discriminate, or show prejudice
towards a particular group. In this case, options a, b, and c contain biases towards people’s physical
abilities (height), hearing ability, and age, respectively. However, "I think I’ve been gypped" is an
informal expression meaning to be cheated or deceived, which, while potentially offensive, does not
directly target a specific group based on attributes like race, gender, or disabilities.
Figure A4: The refined version of the input presents the question and answers in a clearer, more
structured format that improves readability and comprehension. Specifically, using a bulleted list
for the options makes them distinct and easier to evaluate individually. Moreover, the addition of
an explanation provides context and justification for the selected answer, enhancing the educational
value of the content. This refined structure and the inclusion of an explanatory note make it a superior
training example for models, as it helps in understanding not just the answer but the rationale behind
it, promoting deeper learning and better pattern recognition in language processing tasks.
21


Good Case 5
Extracted:
Question: I am stucked on the following challenge: "If the line determined by two distinct points
(x1, y1) and (x2, y2) is not vertical, and therefore has slope (y2 −y1)/(x2 −x1), show that the point-
slope form of its equation is the same regardless of which point is used as the given point." Okay, we
can separate (x0, y0) from the form to get:
y(x2 −x1) −x(y2 −y1) = y0(x2 −x1) −x0(y2 −y1)
But how exclude this point (x0, y0) and leave only x, y, x1, y1, x2, y2 in the equation? UPDATE: There
is a solution for this challenge:
(y1 −y2)x + (x2 −x1)y = x2y1 −x1y2
From the answer I found that
y2(x −x1) −y1(x −x2) = y(x2 −x1)
... but why this is true?
Answer: Thanks to saulspatz, the solution is to simply show that, whether we are using (x1, y1) or
(x2, y2) as the given point, the equation does not change.
So both equations:
y −y1 = m(x −x1)
y −y2 = m(x −x2)
reduce to:
(y1 −y2)x + (x2 −x1)y = x2y1 −x1y2.
Refined:
Question: Demonstrate that the point-slope form of the equation of a line determined by two distinct
points (x1, y1) and (x2, y2), where the line is not vertical, remains consistent regardless of which point
is chosen as the reference. Why does the form of the equation not change when different reference
points are used?
Answer: To address this challenge, we need to confirm that the point-slope equation of a line remains
consistent regardless of the point chosen to derive it. Given two points (x1, y1) and (x2, y2) with a
non-vertical line passing through them, the slope m of the line is y2−y1
x2−x1 .
The point-slope form of a line’s equation based on point (x1, y1) is:
y −y1 = m(x −x1)
Similarly, using point (x2, y2) gives:
y −y2 = m(x −x2)
These can be rewritten and compared to show their equivalence. Expanding both equations: - From the
first equation: y −y1 = y2−y1
x2−x1 (x −x1) - From the second equation: y −y2 = y2−y1
x2−x1 (x −x2)
We simplify these to find a common form. Multiplying both sides by (x2 −x1) and rearranging terms,
we find:
y2(x −x1) −y1(x −x2) = y(x2 −x1)
This confirms that both forms collapse to the same linear equation, showing the consistency of the
point-slope form irrespective of the chosen reference point. Thus, it is demonstrated that the structural
form of the equation remains unchanged, ensuring that the mathematical properties of the line—defined
by its slope and points through which it passes—are invariant to the choice of the reference point used
in its expression. This uniformity stems from the linearity and the constant slope defined by any two
points on a straight line.
Figure A5: The original answer contains an irrelevant introductory sentence, which makes it less
focused. In contrast, the refined version provides a clearer question by explicitly stating the problem
of demonstrating the invariance of the point-slope form. The explanation in the answer is expanded
to show step-by-step equivalence between the two forms and includes detailed reasoning about why
the consistency holds. The added explanation reinforces the understanding of linear equations and
the concept of constant slope, making the solution more comprehensive and pedagogically valuable.
22


Bad Case 1
Extracted:
Question: Sort
n
−3π, 9, 17
√
3, −4, 47
7 , 3
e,
3
√
5
o
.
Answer:
n
−3π, −4, 3
e,
3
√
5, 47
7 , 9, 17
√
3
o
.
Refined:
Question: Sort the following numbers in ascending order: −3π, 9, 17
√
3, −4, 47
7 , 3
e,
3
√
5.
Answer: The sorted numbers in ascending order are: −3π, −4, 3
e,
3
√
5, 47
7 , 9, 17
√
3.
Figure A6: The refined example does not enhance the clarity or quality of the information. The
original formatting and question structure were already clear and concise, and the transformation
into a slightly more verbose question-and-answer format does not add significant value. Additionally,
both versions use proper LaTeX formatting for mathematical expressions, so there is no improvement
in readability or technical accuracy. Essentially, the refinement introduces additional words without
enhancing comprehension, which could potentially dilute the effectiveness of the model training by
focusing on unnecessary stylistic alterations rather than content clarity or quality.
Bad Case 2
Extracted:
Question: Which curve (AD or SRAS) will be affected and which direction (leftward or rightward)
will it shift when Congress goes to war against Canada?
Answer: When Congress falls into a war with Canada, the aggregate demand curve will be affected by
moving to the left. Suppose Congress fights against Canada, then, it is likely that Congress will pass
laws that advocate for increased tariffs on imports.
Spending on exports is an element of aggregate demand. Thus, its curve shifts rightwards, when its
components, which are, spending on consumptions, government spending, or spending on investment
increase.
On the other hand, Canada will not do business with the USA as usual when they fall out with Congress.
For example, the expenses on exports to Canada will reduce.
Thus, the aggregate demand will fail to increase and shift on the left if there is less consumption from
households, falls of investments from firms or when the demand for exports from a country like Canada
falls.
Refined:
Question: How does a war between Congress and Canada affect the Aggregate Demand (AD) curve,
and in which direction will it shift?
Answer: When Congress declares war against Canada, the
Aggregate Demand (AD) curve shifts to the left because Congress will implement increased tariffs on
imports. Spending on exports, which is a component of aggregate demand, decreases due to these tariffs.
The Short Run Aggregate Supply (SRAS) curve, however, is not directly affected by this situation. It
mainly reflects changes in production costs and the availability of resources, which are not directly
influenced by international conflicts unless they lead to disruptions in the domestic production process.
In summary, the AD curve shifts left due to decreased exports and potentially lower consumer and
business spending, while the SRAS curve remains relatively stable unless there are indirect impacts on
domestic production.
Figure A7: The refined answer, while more structured and clearer in terms of the specific economic
curves discussed, introduces inaccuracies that were not present in the original. Specifically, the
refined version inaccurately states that spending on exports decreases due to tariffs, which contradicts
the fundamental economic concept that tariffs affect imports rather than exports. Furthermore,
the original answer provides a broader analysis of potential economic impacts, such as changes
in household consumption and firm investment, which are omitted in the refined version. These
omissions reduce the depth and accuracy of the economic analysis provided in response to the effects
of war on economic curves.
23


Bad Case 3
Extracted:
Question: Is there a tool (LaTeX, JavaScript, Mathematica..) that allows one to draw Venn diagram
automatically from subsets relations, e.g. A ⊂A + B, A ⊂C, C ⊂C + D, B ̸⊂C would yield:
|
|
|
A+B
|
|
|
|
|
|
|
|
|
|
A
|
|
|
|
|
|
|
|
C+D
|