benchmarks.wiki / Public workspace
MMLU
Reasoning / Knowledge
MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.
mmlu
Full text: eligible public problems include text, images and discussion.
Reference links
At a glance
What a problem looks like
A question with four options, plus the subject it belongs to. The options are imported as a list; the index of the correct one is not.
How it is scored
The source ships the answer index with every problem; this catalogue keeps it in restricted storage and never renders it.
Metric: accuracy (fraction of questions answered correctly, usually reported as a macro average over the 57 subjects)
Why it is hard
No single subject is hard; the breadth is. A model must hold professional-level material in law, medicine and mathematics at once, and the four-way format leaves little room to recover from a gap in knowledge.
Cite
@article{hendrycks2020mmlu,
title={Measuring Massive Multitask Language Understanding},
author={Dan Hendrycks and Collin Burns and Steven Basart and Andy Zou and Mantas Mazeika and Dawn Song and Jacob Steinhardt},
journal={arXiv preprint arXiv:2009.03300},
year={2020}
}Discussion
Contribute
Share a reproduction, report an evaluation issue, or suggest a correction to this benchmark overview.
Tasks
Whole public benchmark.
- Public problems
- 200
- Formats
- text (200)
- Problems with images
- 0
- Answer availability
- Answer published by the source (200)
- Configs
- all (200)
- Splits
- test (200)
- Awaiting a first discussion
- 200
- Problems with a discussion
- 0
- Public contributions
- 0
Provenance and terms
Pinned source revisions, newest first:
c30699e8356da336a370243923dbaf21066bb9fe· acquired 2026-09-12 · release8cfe6e6995d0
- A possible effect of damage to the third cranial nerve by trauma to the orbit is
Answer published by the source
- Food is prevented from entering the larynx during the pharyngeal phase of…
Answer published by the source
- Which of the following terms describes the motion of bending the forearm toward…
Answer published by the source
- Statement 1 | Every nonzero free abelian group has an infinite number of bases.…
Answer published by the source
- In the group G = 2, 4, 6, 8) under multiplication modulo 10, the identity…
Answer published by the source
- Noradrenaline is the neurotransmitter between which of the two structures below?
Answer published by the source
- The muscles of the soft palate are innervated by branches of the
Answer published by the source
- On which surfaces of the teeth is dental plaque most likely to accumulate in the…
Answer published by the source
- In a patient with a normal healthy mouth, you would expect the mucosa covering…
Answer published by the source
- Laryngeal oedema is usually confined to the larynx above the vocal folds because
Answer published by the source
- Determine whether the polynomial in Z[x] satisfies an Eisenstein criterion for…
Answer published by the source
- Which of one of the following statements about the postnatal growth of the…
Answer published by the source
- Statement 1 | The image of a group of 6 elements under a homomorphism may have…
Answer published by the source
- Compute the product in the given ring. (12)(16) in Z_24
Answer published by the source
- Calcium, potassium, and sodium are classified as which of the following?
Answer published by the source
- Statement 1 | Every maximal ideal is a prime ideal. Statement 2 | If I is a…
Answer published by the source
- Statement 1 | Every quotient ring of every commutative ring is again a…
Answer published by the source
- Find the characteristic of the ring Z_3 x 3Z.
Answer published by the source
- The anatomic location of the spinal canal is
Answer published by the source
- Statement 1 | A ring homomorphism is one to one if and only if the kernel is…
Answer published by the source
- The inverse of -i in the multiplicative group, 1, -1, i , -i is
Answer published by the source
- Which of the following is the lymphoid organ that is a reservoir for red blood…
Answer published by the source
- Which of the following cavities are separated by the diaphragm?
Answer published by the source
- A subset H of a group (G,*) is a group if
Answer published by the source
- Which muscles are normally active during quiet inspiration?
Answer published by the source
- Statement 1 | Every group of order 159 is cyclic. Statement 2 | Every group of…
Answer published by the source
- Find the degree for the given field extension Q(sqrt(2)) over Q.
Answer published by the source
- The auditory tube
Answer published by the source
- Statement 1 | If H is a subgroup of G and a belongs to G then |aH| = |Ha|.…
Answer published by the source
- The physician directs the medical assistant to complete a request form for an…
Answer published by the source
- Statement 1 | For every positive integer n there is a cyclic group of order n.…
Answer published by the source
- Statement 1 | 4x - 2 is irreducible over Z. Statement 2 | 4x - 2 is irreducible…
Answer published by the source
- The crown of a very carious tooth suddenly fractures while the patient is…
Answer published by the source
- The phrenic nerves innervate the
Answer published by the source
- Statement 1 | If H and K are subgroups of a group G, then |HK| = |H||K|/|H…
Answer published by the source
- If A = (1, 2, 3, 4). Let ~= (1, 2), (1, 3), (4, 2). Then ~ is
Answer published by the source
- Find the characteristic of the ring Z x Z.
Answer published by the source
- When you examine a patient who has suffered an unfavourable fracture of the body…
Answer published by the source
- The walls of the maxillary sinus are sinus is related to
Answer published by the source
- Statement 1 | If R is a ring and f(x) and g(x) are in R[x], then deg (f(x)+g(x))…
Answer published by the source
- Which of the following substances is found in greater quantity in exhaled air?
Answer published by the source
- The set of all real numbers under the usual multiplication operation is not a…
Answer published by the source
- Statement 1 | If a R is an integral domain, then R[x] is an integral domain.…
Answer published by the source
- Using Fermat's theorem, find the remainder of 3^47 when it is divided by 23.
Answer published by the source
- Determine whether the polynomial in Z[x] satisfies an Eisenstein criterion for…
Answer published by the source
- Which of the following describes the cluster of blood capillaries found in each…
Answer published by the source
- The walls of bronchioles contain
Answer published by the source
- Which of the following muscles initiates elevation of the mandible?
Answer published by the source
- The polynomial x^4 + 4 can be factored into linear factors in Z_5[x]. Find this…
Answer published by the source
- Statement 1 | A factor group of a non-Abelian group is non-Abelian. Statement 2…
Answer published by the source