Discussion
0 posts
No posts in this thread yet.
benchmarks.wiki / Public workspace
MATH
Mathematics
MATH organises 12,500 high-school competition problems into seven subjects and five difficulty levels, each with a full worked solution. It underlies much of the later mathematics evaluation work; this catalogue imports the full test split: 5,000 problems across all seven subjects.
math
Full text: eligible public problems include text, images and discussion.
History
Dan Hendrycks and collaborators introduced MATH in 2021 to test mathematical problem solving beyond routine word problems. They assembled competition mathematics with worked solutions so that evaluation could address harder reasoning and training could use intermediate steps. EleutherAI maintains the Hugging Face distribution recorded here, whose 2023 publication date is later than the original benchmark.
How it was built
MATH collects 12,500 high-school competition problems with worked solutions, organised into seven subjects and five difficulty levels. The material comes from existing competition problem collections rather than newly generated model outputs. The catalogue contains a 200-problem selection from the algebra test split, whose source contains 1,187 problems; that selection is not the full benchmark.
At a glance
What a problem looks like
A problem statement in LaTeX, with its subject and difficulty level. The worked solution is not part of the prompt.
How it is scored
The source ships a step-by-step solution with every problem; this catalogue keeps it in restricted storage and never renders it.
Metric: accuracy on the final boxed answer
Reported scores
Evaluation conditions differ between rows, so this is not a ranking. Rows marked vendor are reports by the model’s own organization.
| Model | Score | Metric | Setup | Reported by | Source |
|---|---|---|---|---|---|
| DeepSeek-V3DeepSeek | 61.6 | EM | Base model; 4-shot. | vendor (self-reported) | Base Model, Standard Benchmarks, MATH (EM), final column |
Why it is hard
The problems are written for students who have the technique and need to choose it. Difficulty is stated per problem, so level 5 items are hard by construction rather than by accident.
Limitations and criticism
Final-answer accuracy does not establish that the reasoning leading to an answer is valid. Answer extraction and equivalence checking can affect whether mathematically equivalent responses receive credit. Public problems and worked solutions create a risk of training exposure. Results on an algebra selection or another reduced subset should be distinguished from results on the full test set.
How much room is left
The recorded vendor report gives DeepSeek-V3 base 61.6 percent exact match with four examples in the prompt. This leaves considerable headroom for that setup, but one base-model report is insufficient to establish the ceiling for stronger reasoning systems. Comparisons need to identify the evaluated subset and whether the model received tools or multiple attempts.
Reference links
Cite
@article{hendrycks2021math,
title={Measuring Mathematical Problem Solving With the MATH Dataset},
author={Dan Hendrycks and Collin Burns and Saurav Kadavath and Akul Arora and Steven Basart and Eric Tang and Dawn Song and Jacob Steinhardt},
journal={arXiv preprint arXiv:2103.03874},
year={2021}
}Discussion
1 thread and 0 posts on this page. Anyone may reply; no account is required.
Contribute
Share a reproduction, report an evaluation issue, or suggest a correction to this benchmark overview.
Tasks
Whole public benchmark.
- Public problems
- 5,000
- Formats
- math (5,000)
- Problems with images
- 0
- Answer availability
- Answer published by the source (5,000)
- Configs
- algebra (1,187) · counting_and_probability (474) · geometry (479) · intermediate_algebra (903) · number_theory (540) · prealgebra (871) · precalculus (546)
- Splits
- test (5,000)
- Awaiting a first discussion
- 5,000
- Problems with a discussion
- 0
- Public contributions
- 0
Provenance and terms
Pinned source revisions, newest first:
21a5633873b6a120296cce3e2df9d5550074f4a3· acquired 2026-09-23 · release36bf50fafb0221a5633873b6a120296cce3e2df9d5550074f4a3· acquired 2026-09-12 · releaseae4d22a7a5ac21a5633873b6a120296cce3e2df9d5550074f4a3· acquired 2026-09-12 · release7febcb3508c0
- Melanie ran at 5 m/s for 400 meters, then ran at 4 m/s for the next 800 meters, and finally ran at 8 m/s for the final 400 meters. What was her average speed, in meters per second,…
Answer published by the source
- How much greater than 27² is 43² ?
Answer published by the source
- Given the functions f(x)=(x+5)/(3) and g(x)=(1)/(f^(−1)(x)+1), find the value of g(3).
Answer published by the source
- Point A lies somewhere within or on the square which has opposite corners at (0,0) and (2,2). Point B lies somewhere within or on the square which has opposite corners at points …
Answer published by the source
- The difference of the roots of the quadratic equation x²+bx+c=0 is |b−2c|. If c≠0, then find c in terms of b.
Answer published by the source
- Rationalize the denominator of (2)/(root[3](4)+root[3](32)). The answer can be written in the form of (root[3](A))/(B), where A and B are positive integers. Find the minimum possib…
Answer published by the source
- How many different values are represented by the following quantities? 3^(−2),9^(−2),27^(−(2)/(3)),9⋅81^(−1),243^(−(4)/(5))
Answer published by the source
- If displaystyle{f(x)=x^{(x+1)}(x+2)^{(x+3)}}, then find the value of f(0)+f(−1)+f(−2)+f(−3).
Answer published by the source
- What is the 2003rd term of the sequence of odd numbers 1, 3, 5, 7, …?
Answer published by the source
- If x, y, and z are real numbers for which …xyz?
Answer published by the source
- Let f(x) = left{
begin{array}{cl}
2x + 1 & text{if } x le 3,
8 - 4x & text{if } x > 3.
end{array}
right.Find the sum of all values of x such that f(x)=0.
Answer published by the source
- There are 360 people in my school. 15 take calculus, physics, and chemistry, and…
Answer published by the source
- If we flip 6 coins, what is the probability of getting at least 2 heads?
Answer published by the source
- In the row of Pascal's triangle that starts with 1 and then 10, what is the next…
Answer published by the source
- Mikka wants to order a pizza with two different toppings. He has 8 different…
Answer published by the source
- Two jokers are added to a 52 card deck and the entire stack of 54 cards is shuffled randomly. What is the expected number of cards that will be strictly between the two jokers?
Answer published by the source
- How many different triangles can be formed using three vertices of a hexagon as…
Answer published by the source
- Mr. and Mrs. Lopez have three children. When they get into their family car, two…
Answer published by the source
- In the diagram below, we have sin∠RPQ=(7)/(25). What is cos∠RPS? [asy] pair R,P,Q,SS; SS = (-2,0); P = (0,0); Q = (2,0); R = rotate(aSin(7/25))*(1.5,0); dot("S",SS,S); dot("Q",Q,S)…
Answer published by the source
- In △XYZ, ∠Z=90^(∘), cosY=(7)/(25), and XY=25. What is tanX?
Answer published by the source
- In convex hexagon ABCDEF, all six sides are congruent, ∠A and ∠D are right angles, and ∠B, ∠C, ∠E, and ∠F are congruent. The area of the hexagonal region is 2116(√(2)+1). Find AB.
Answer published by the source
- An equiangular octagon has four sides of length 1 and four sides of length (√(2))/(2), arranged so that no two consecutive sides have the same length. What is the area of the octag…
Answer published by the source
- A water-filled spherical balloon lands on a sidewalk, momentarily flattening to…
Answer published by the source
- A company makes a six-sided hollow aluminum container in the shape of a…
Answer published by the source
- Let P be a point on the hyperbola x²+8xy+7y²=225. Find the shortest possible distance from the origin to P.
Answer published by the source
- Let z and w be complex numbers such that |z|=2 and |w|=5. Find the largest possible value of |z+w|.
Answer published by the source
- Find the infinite sum of
Answer published by the source
- Let p(x) be a polynomial with positive leading coefficient such that
Answer published by the source
- Let a and b be real numbers such that a>2b>0. Find the minimum value of 3a³√(3)+(2)/(ab−2b²).
Answer published by the source
- Find the sum of the real roots of x⁴−80x−36=0.
Answer published by the source
- Find the minimum value of the function
Answer published by the source
- The range of the function f(x) is [−3,5]. Let g(x)=[f(x)]². Find the range of the function g(x).
Answer published by the source
- Compute
Answer published by the source
- At a competition with N players, the number of players given elite status is equal to 2^(1+⌊log₂(N−1)⌋)−N. Suppose that 19 players are given elite status. What is the sum of the tw…
Answer published by the source
- If log₃6≈1.63, what is the value of log₉4, accurate to two decimal places?
Answer published by the source
- Determine the complex number z satisfying the equation 3z+4i(z)¯=1−8i. Note that (z)¯ denotes the conjugate of z.
Answer published by the source
- For how many positive integers m, 1≤m≤1000, does there exist at least one positive integer n such that m⋅n≤m+n?
Answer published by the source
- Find the real roots of
Answer published by the source
- What is the residue modulo 13 of the sum of the modulo 13 inverses of the first 12 positive integers? Express your answer as an integer from 0 to 12, inclusive.
Answer published by the source
- What integer n satisfies 0≤n<101 and 123456equiv npmod {101}~?
Answer published by the source
- What is the value of a+b+c+d+e+f for the decimal representation of (4)/(37)+(3)/(11)+(23)/(9)=2.abcdef…?
Answer published by the source
- The five-digit number N=14,9AB is divisible by 12. Given that A and B are nonzero digits, what is the smallest possible value of N?
Answer published by the source
- The table shows some of the results of a survey by radio station KAMC. What…
Answer published by the source
- Simplify √(32670).
Answer published by the source
- What is the greatest common divisor of 1313 and 1001?
Answer published by the source
- How many positive integers are less than 200π?
Answer published by the source
- If |z|=5, and |w|=2, what is the minimum value of |z−w|?
Answer published by the source
- Find all values of k, for which there exists a nonzero vector v such that begin{pmatrix} 2 & -2 & 1 2 & -3 & 2 -1 & 2 & 0 end{pmatrix} mathbf{v} = k mathbf{v}.
Answer published by the source
- Find the matrix that corresponds to rotating about the origin by an angle of…
Answer published by the source
- A line is parameterized by
Answer published by the source