Discussion
0 posts
No posts in this thread yet.
benchmarks.wiki / Public workspace
Omni-MATH
Mathematics
Omni-MATH evaluates mathematical reasoning on Olympiad-level problems. Its official dataset contains 4,428 problems accompanied by domain and difficulty information.
omni-math
Full text: eligible public problems include text, images and discussion.
History
The Omni-MATH project introduced its Olympiad-level mathematics benchmark in 2024, with this catalogue recording a September Hugging Face publication under KbsdJames. It targeted mathematical reasoning on difficult competition problems. Domain and difficulty annotations support analysis beyond a single aggregate accuracy figure.
How it was built
The official dataset contains 4,428 Olympiad-level mathematical problems accompanied by domain and difficulty information. The distribution supplies answers, and the authors also publish an evaluator called Omni-Judge. The catalogue uses a capped selection of 200 task records rather than the entire collection.
At a glance
What a problem looks like
Olympiad-level mathematical problem statements; the generic import accepts the scalar problem field.
How it is scored
The source ships an answer with every problem; this catalogue keeps it in restricted storage and never renders it. The authors publish Omni-Judge, an open evaluator for this dataset, at https://huggingface.co/KbsdJames/Omni-Judge.
Why it is hard
Requires mathematical reasoning on Olympiad-level problems.
Limitations and criticism
Olympiad-level mathematics is a specialised domain and does not directly measure mathematical modelling, empirical work or formal proof verification. A correct final answer alone is not evidence that every step of a solution is valid. Reports need to identify the evaluator and any tool or sampling budget, since those choices affect what a success rate represents.
Reference links
Cite
@misc{gao2024omnimathuniversalolympiadlevel,
title={Omni-MATH: A Universal Olympiad Level Mathematic Benchmark For Large Language Models},
author={Bofei Gao and Feifan Song and Zhe Yang and Zefan Cai and Yibo Miao and Qingxiu Dong and Lei Li and Chenghao Ma and Liang Chen and Runxin Xu and Zhengyang Tang and Benyou Wang and Daoguang Zan and Shanghaoran Quan and Ge Zhang and Lei Sha and Yichang Zhang and Xuancheng Ren and Tianyu Liu and Baobao Chang},
year={2024},
eprint={2410.07985},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2410.07985},
}Discussion
1 thread and 0 posts on this page. Anyone may reply; no account is required.
Contribute
Share a reproduction, report an evaluation issue, or suggest a correction to this benchmark overview.
Tasks
Whole public benchmark.
- Public problems
- 4,406
- Formats
- math (4,206) · text (200)
- Problems with images
- 0
- Answer availability
- Answer published by the source (4,406)
- Configs
- Default (4,406)
- Splits
- test (4,406)
- Awaiting a first discussion
- 4,406
- Problems with a discussion
- 0
- Public contributions
- 0
Provenance and terms
Pinned source revisions, newest first:
40ba231d8f16e29ecd40e6407e2c8640145a8f62· acquired 2026-09-09 · release2ee23763677440ba231d8f16e29ecd40e6407e2c8640145a8f62· acquired 2026-09-09 · release27a71f2fd34a40ba231d8f16e29ecd40e6407e2c8640145a8f62· acquired 2026-09-09 · release9933a682d06440ba231d8f16e29ecd40e6407e2c8640145a8f62· acquired 2026-09-09 · releasee02b298818a840ba231d8f16e29ecd40e6407e2c8640145a8f62· acquired 2026-09-08 · released2fdbb3503e8
- Five identical empty buckets of 2-liter capacity stand at the vertices of a regular pentagon. Cinderella and her wicked Stepmother go through a sequence of rounds: At the beginning…
Answer published by the source
- Lucas writes two distinct positive integers on a whiteboard. He decreases the…
Answer published by the source
- A small village has n people. During their yearly elections, groups of three people come up to a stage and vote for someone in the village to be the new leader. After every possibl…
Answer published by the source
- How many different graphs with 9 vertices exist where each vertex is connected…
Answer published by the source
- Find the largest possible integer k, such that the following statement is true: Let 2009 arbitrary non-degenerated triangles be given. In every triangle the three sides are coloure…
Answer published by the source
- Let ABCD be a tetrahedron such that its circumscribed sphere of radius R and its inscribed sphere of radius r are concentric. Given that AB=AC=1≤BC and R=4r, find BC².
Answer published by the source
- How many six-digit multiples of 27 have only 3, 6, or 9 as their digits?
Answer published by the source
- Find the maximum number of points X_(i) such that for each i, triangle A B X_{i} cong triangle C D X_{i}.
Answer published by the source
- Compute the number of integers n∈{1,2,…,300} such that n is the product of two distinct primes, and is also the length of the longest leg of some nondegenerate right triangle with …
Answer published by the source
- This question is unrelated to the graph shown in part a; instead, we consider a…
Answer published by the source
- Fix a triangle ABC. We say that triangle XYZ is elegant if X lies on segment BC, Y lies on segment CA, Z lies on segment AB, and XYZ is similar to ABC (i.e., ∠A=∠X,∠B=∠Y,∠C=∠Z). Of…
Answer published by the source
- What is 25% of 60?
Answer published by the source
- Two unit squares S₁ and S₂ have horizontal and vertical sides. Let x be the minimum distance between a point in S₁ and a point in S₂, and let y be the maximum distance between a po…
Answer published by the source
- Compute the number of ordered pairs of integers (x,y) such that x²+y²<2019 and x²+min(x,y)=y²+max(x,y)
Answer published by the source
- Determine whether √(1001²+1)+√(1002²+1)+⋯+√(2000²+1) be a rational number or not?
Answer published by the source
- What is the value of the expression 4+(3)/(10)+(9)/(1000)?
Answer published by the source
- How many points does a sports team earn for 9 wins, 3 losses, and 4 ties, if…
Answer published by the source
- If wxyz is a four-digit positive integer with w≠0, the layer sum of this integer equals wxyz+xyz+yz+z. If the layer sum of wxyz equals 2014, what is the value of w+x+y+z?
Answer published by the source
- Find all integers n satisfying n≥2 and (σ(n))/(p(n)−1)=n, in which σ(n) denotes the sum of all positive divisors of n, and p(n) denotes the largest prime divisor of n.
Answer published by the source
- If x=2, what is the value of 4x²−3x²?
Answer published by the source
- If 2×2×3×3×5×6=5×6×n×n, what is a possible value of n?
Answer published by the source
- Determine all pairs (x,y) of positive integers such that x²y+x+y is divisible by xy²+y+7.
Answer published by the source
- While Travis is having fun on cubes, Sherry is hopping in the same manner on an…
Answer published by the source
- We colour all the sides and diagonals of a regular polygon P with 43 vertices either red or blue in such a way that every vertex is an endpoint of 20 red segments and 22 blue segme…
Answer published by the source
- Let ABC be a fixed acute triangle inscribed in a circle ω with center O . A variable point X is chosen on minor arc AB of ω , and segments CX and AB meet at D . Denote by O₁ and O₂…
Answer published by the source
- Storage space on a computer is measured in gigabytes (GB) and megabytes (MB),…
Answer published by the source
- Evaluate the sum …lnx denotes the natural logarithm of x.)
Answer published by the source
- There is a city with n citizens. The city wants to buy [i]sceptervirus[/i] tests with which it is possible to analyze the samples of several people at the same time. The result of …
Answer published by the source
- The country Dreamland consists of 2016 cities. The airline Starways wants to…
Answer published by the source
- Kelvin the Frog has a pair of standard fair 8-sided dice (each labelled from 1…
Answer published by the source
- Evaluate ∑_(k=1)^(∞)((−1)^(k−1))/(k)∑_(n=0)^(∞)(1)/(k2^(n)+1).
Answer published by the source
- The real numbers x,y and z satisfy the three equations x+y=7, xz=−180, and (x+y+z)²=4. If S is the sum of the two possible values of y, what is −S?
Answer published by the source
- Mark writes the expression √(d) for each positive divisor d of 8 ! on the board. Seeing that these expressions might not be worth points on HMMT, Rishabh simplifies each expression…
Answer published by the source
- Let ABCD be an isosceles trapezoid with AB=1,BC=DA=5,CD=7. Let P be the intersection of diagonals AC and BD, and let Q be the foot of the altitude from D to BC. Let PQ intersect AB…
Answer published by the source
- Determine the value of ∑_(n=1)^(∞)ln(1+(1)/(n))⋅ln(1+(1)/(2n))⋅ln(1+(1)/(2n+1)).
Answer published by the source
- Alice starts with the number 0. She can apply 100 operations on her number. In…
Answer published by the source
- Find the number of integers n such that 1+leftlfloorfrac{100 n}{101}rightrfloor=leftlceilfrac{99 n}{100}rightrceil
Answer published by the source
- Let n≥2 be a natural. Define
Answer published by the source
- What is the maximum number of colours that can be used to paint an 8×8 chessboard so that every square is painted in a single colour, and is adjacent , horizontally, vertically but…
Answer published by the source
- Two different integers u and v are written on a board. We perform a sequence of steps. At each step we do one of the following two operations: (i) If a and b are different integers…
Answer published by the source
- Minimize the function F(x₁,x₂,⋯,x_(n))=∑_(i=1)^(n)|x_(i)| subject to the condition that after redistribution we should have at each A_(i),a_(i)−x₁+x_(i−1)=N for i∈{1,2,…,n} where …
Answer published by the source
- Consider an isosceles triangle T with base 10 and height 12. Define a sequence ω₁,ω₂,… of circles such that ω₁ is the incircle of T and ω_(i+1) is tangent to ω_(i) and both legs of…
Answer published by the source
- Let P and Q be points on line l with PQ=12. Two circles, ω and Ω, are both tangent to l at P and are externally tangent to each other. A line through Q intersects ω at A and B, wit…
Answer published by the source
- Let n>1 be a positive integer. Each unit square in an n×n grid of squares is colored either black or white, such that the following conditions hold: - Any two black squares can be …
Answer published by the source
- How many positive integers at most 420 leave different remainders when divided…
Answer published by the source
- Find all the triples of positive integers (a,b,c) for which the number
Answer published by the source
- Fifty numbers have an average of 76. Forty of these numbers have an average of…
Answer published by the source
- Abbot writes the letter A on the board. Every minute, he replaces every occurrence of A with AB and every occurrence of B with BA, hence creating a string that is twice as long. Af…
Answer published by the source
- For each positive integer n , find the number of n -digit positive integers that satisfy both of the following conditions: • no two consecutive digits are equal, and • the last dig…
Answer published by the source
- In 12 years, Janice will be 8 times as old as she was 2 years ago. How old is…
Answer published by the source