Discussion
0 posts
No posts in this thread yet.
benchmarks.wiki / Public workspace
MMLU-Pro
Reasoning
MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.
mmlu-pro
Full text: eligible public problems include text, images and discussion.
History
TIGER-Lab introduced MMLU-Pro in 2024 to make broad academic question answering more demanding as the original MMLU became less discriminating. The revision emphasised harder reasoning questions and expanded the answer choices from four to ten. Its Hugging Face publication is recorded as May 2024.
How it was built
MMLU-Pro rebuilds the question collection by removing easier original items and using expanded sets of answer options. It covers fourteen academic and professional subjects in a multiple-choice format. The catalogue imports 200 test questions, a capped selection that should not be treated as the size of the complete benchmark.
At a glance
What a problem looks like
Multiple-choice questions with ten options. The import accepts question, options, category and src.
How it is scored
The source ships an answer with every problem; this catalogue keeps it in restricted storage and never renders it.
Reported scores
Evaluation conditions differ between rows, so this is not a ranking. Rows marked vendor are reports by the model’s own organization.
| Model | Score | Metric | Setup | Reported by | Source |
|---|---|---|---|---|---|
| DeepSeek-V3DeepSeek | 64.4 | Acc. | Base model; 5-shot. | vendor (self-reported) | Base Model, Standard Benchmarks, MMLU-Pro (Acc.), final column |
Why it is hard
Ten options rather than four, and the easiest original items removed, so guessing and memorisation carry much less of the score.
Limitations and criticism
Ten options reduce the probability of uniform random guessing but do not prevent memorisation or exploitation of answer cues. A correct choice does not directly validate a model's explanation or its ability to solve an open-ended version of the task. Subject composition, prompting and answer extraction remain relevant to comparisons, and reuse of public question material leaves a risk of training exposure.
How much room is left
The recorded vendor report gives DeepSeek-V3 base 64.4 percent accuracy with five examples in the prompt. That result remains well below the ceiling for the reported setup and supports treating the test as challenging for that model. It does not establish the current leading score or the remaining headroom for every model family.
Reference links
Discussion
1 thread and 0 posts on this page. Anyone may reply; no account is required.
Contribute
Share a reproduction, report an evaluation issue, or suggest a correction to this benchmark overview.
Tasks
Whole public benchmark.
- Public problems
- 12,032
- Formats
- text (12,032)
- Problems with images
- 0
- Answer availability
- Answer published by the source (12,032)
- Configs
- default (12,032)
- Splits
- test (12,032)
- Awaiting a first discussion
- 12,032
- Problems with a discussion
- 0
- Public contributions
- 0
Provenance and terms
Pinned source revisions, newest first:
b189ec765aa7ed75c8acfea42df31fdae71f97be· acquired 2026-09-11 · releasef17f6fbd935bb189ec765aa7ed75c8acfea42df31fdae71f97be· acquired 2026-09-11 · release518b10e2176cb189ec765aa7ed75c8acfea42df31fdae71f97be· acquired 2026-09-11 · releasec2f53d6ad184
- In the diamond structure of elemental carbon, the nearest neighbors of each C…
Answer published by the source
- An eastbound car travels a distance of 40 m in 5 s. Determine the speed of the…
Answer published by the source
- The access matrix approach to protection has the difficulty that
Answer published by the source
- A programmer wrote the program below. The program uses a list of numbers called…
Answer published by the source
- Construct a complete truth table for the following argument. Then, using the…
Answer published by the source
- Shapiro claims that surveys have been confirmed by longitudinal studies that…
Answer published by the source
- An electromagnetic wave of40KHzis radiated by an antenna . Find the time taken…
Answer published by the source
- A low loss transmission line, operating at a frequency of 400MHz, with air as…
Answer published by the source
- Consider a vertical cylinder which has an outside diameter of 1.0 ft and is 2.0…
Answer published by the source
- Which of the following would yield the greatest net return to acorporation in…
Answer published by the source
- An air-coretoroidhas 500 turns, a cross-sectional area of 6 cm^2, a mean radius…
Answer published by the source
- Estimate the pressure which would be exerted by 8.2 lb of CO in a 1-ft^3…
Answer published by the source
- Is extra-territorial abduction a ground for refusal of otherwise lawful…
Answer published by the source
- Which reason may constitute sufficient good cause or excusable neglect to enable…
Answer published by the source
- A man decided to stop at a drive-through hamburger stand for a late snack. As he…
Answer published by the source
- Which of the following is not within the power of a federal subpoena to command…
Answer published by the source
- A landlord, an owner in fee simple, leased a three- story house to a tenant for…
Answer published by the source
- A defendant was arrested and charged with possession of heroin. At the…
Answer published by the source
- Mechanical energy is transduced to electrochemical energy by
Answer published by the source
- (A) How many gallons are there in 12 liters? (B) How many liters are there in 10…
Answer published by the source
- A 38-year-old person is afraid to travel on public transportation. A program of…
Answer published by the source
- A child with type A B blood inherits an IA allele from one parent and an IB…
Answer published by the source
- Are the daughter nuclei formed during cleavage exactly equivalent , or is there…
Answer published by the source
- Iron (III) oxide is reacted with carbon monoxide in a blast furnace to produce…
Answer published by the source
- Calculate the percentage difference between e^(x) and 1+x for x=0.0050
Answer published by the source
- A buffer is made from equal concentrations of a weak acid and its conjugate…
Answer published by the source
- Calculate the recoil energy of an atom of mass 10, following emission of a…
Answer published by the source
- Calculate the characteristicvibrationalfrequency of the lowest vibration state…
Answer published by the source
- Machu Picchu was built in the mountains as a:
Answer published by the source
- The possibility that Homo erectus possessed an aesthetic sense and may have…
Answer published by the source
- According to The Federalist Papers, federalism has which of the following…
Answer published by the source
- About what percentage of survey respondents from India report having paid a…
Answer published by the source
- Which brand of cat food claims it's so tasty that 'cats ask for it by name'?
Answer published by the source
- Which company holds an annual self-named 'Bake-Off'?
Answer published by the source
- Which diseases are not less frequent among vegetarians?
Answer published by the source
- A deficiency of which vitamin has been associated with enamel defects and…
Answer published by the source
- Which of the following statements about the global ageing population is correct?
Answer published by the source
- Give the significance of the equilibrium level of output.
Answer published by the source
- Express the tax rates (A) 3.06% (B) 6.85% as mills per dollar of assessed…
Answer published by the source
- If real GNP is expanding at a steady annual rate of 2 percent andthe nominal…
Answer published by the source
- Rewrite the expression 6j²−4j+12 in the form c(j+p)²+q, where c,…
Answer published by the source
- Mr. Lucci put together 5 bags of pens. He put 19 black pens and 12 red pens in…
Answer published by the source
- A small kiosk at the Atlanta airport carries souvenirs in the price range of…
Answer published by the source
- On July 7, Magee Data stock sold at a high of 23(1/8) and a low of 22(5/8).…
Answer published by the source
- Find the value of (I / I_0) for the first diagonal maximum of the…
Answer published by the source
- An inventor developed a prototype of an automobile. engine that she believed was…
Answer published by the source
- The sodium doublet has wavelength of 589.0 nm and 589.6 nm. If the diffraction…
Answer published by the source
- The plates of a capacitor are charged to a potential difference of 5 V. If the…
Answer published by the source
- Consider a damped harmonic oscillator. After four cycles the amplitude of the…
Answer published by the source
- If you lived on Venus what phases of the Earth would you see? (Assume your…
Answer published by the source