benchmarks.wiki

benchmarks.wiki / Public workspace

MMLU

Reasoning / Knowledge

MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.

mmlu

Full text14,015 tasks

Official source

Full text: eligible public problems include text, images and discussion.

Download answersBrowse the problemsAnswersOpen a random oneProblems awaiting a first attempt

History

Dan Hendrycks and collaborators introduced MMLU in 2020 to test how broadly language models could apply knowledge across academic and professional subjects. The benchmark spans 57 subjects, from elementary material to professional examinations. Its motivation was to evaluate breadth and depth of learned knowledge beyond the narrower tasks common in earlier language-understanding benchmarks.

How it was built

MMLU collects four-option multiple-choice questions from existing educational and examination materials across 57 subjects. Each item has a subject label and a reference answer index. The source test selection contains 14,042 questions, while this catalog imports only 200 of them. Accuracy measures correct answer selection, with results commonly summarized across subjects.

At a glance

Metric
accuracy (fraction of questions answered correctly, usually reported as a macro average over the 57 subjects)
Size
14,015 unique problems imported from the all / test selection of 14,042 rows; 27 source rows repeat an earlier question exactly and are recorded as variants. The 57 subject configurations overlap this selection and are not imported separately.
License
mit (declared on the Hugging Face dataset card)
Released
Dataset published on Hugging Face 2022-03
Maintainer
cais (source organisation)
Answer access
Columns outside the prompt allowlist, including any answer or reference solution, are separated into restricted storage and never rendered.
Highest recorded
87.1 Acc. — DeepSeek-V3 (vendor, self-reported)

What a problem looks like

A question with four options, plus the subject it belongs to. The options are imported as a list; the index of the correct one is not.

Read an actual problem

How it is scored

The source ships the answer index with every problem; this catalogue keeps it in restricted storage and never renders it.

Metric: accuracy (fraction of questions answered correctly, usually reported as a macro average over the 57 subjects)

Reported scores

Evaluation conditions differ between rows, so this is not a ranking. Rows marked vendor are reports by the model’s own organization.

ModelScoreMetricSetupReported bySource
DeepSeek-V3DeepSeek87.1Acc.Base model; 5-shot.vendor (self-reported)Base Model, Standard Benchmarks, MMLU (Acc.), final column

Why it is hard

No single subject is hard; the breadth is. A model must hold professional-level material in law, medicine and mathematics at once, and the four-way format leaves little room to recover from a gap in knowledge.

Limitations and criticism

The multiple-choice format permits guessing and does not require a model to justify its answer or perform the corresponding professional task. Publicly available source materials and benchmark questions create a risk of overlap with training data, although overlap alone does not establish that a particular model memorized the test. MMLU has also attracted criticism over ambiguous questions and incorrect reference answers. Prompting, answer extraction and subject aggregation can affect comparisons between reports.

How much room is left

The recorded vendor result for the DeepSeek-V3 base model reports 87.1% accuracy in a five-shot setup. This is a high score but still leaves errors on a substantial minority of questions, and one entry is insufficient to establish the benchmark's current ceiling. As results rise, remaining label ambiguities and differences in evaluation procedure become more consequential when interpreting small score gaps.

Cite

@article{hendrycks2020mmlu,
  title={Measuring Massive Multitask Language Understanding},
  author={Dan Hendrycks and Collin Burns and Steven Basart and Andy Zou and Mantas Mazeika and Dawn Song and Jacob Steinhardt},
  journal={arXiv preprint arXiv:2009.03300},
  year={2020}
}

Discussion

1 thread and 0 posts on this page. Anyone may reply; no account is required.

Contribute

Share a reproduction, report an evaluation issue, or suggest a correction to this benchmark overview.

Tasks

Whole public benchmark.

Public problems
14,015
Formats
text (14,015)
Problems with images
0
Answer availability
Answer published by the source (14,015)
Configs
all (14,015)
Splits
test (14,015)
Awaiting a first discussion
14,015
Problems with a discussion
0
Public contributions
0

Provenance and terms

Pinned source revisions, newest first:

  • c30699e8356da336a370243923dbaf21066bb9fe · acquired 2026-09-12 · release 81fc4de3e8e3
  • c30699e8356da336a370243923dbaf21066bb9fe · acquired 2026-09-12 · release 8cfe6e6995d0
Licence
Not recorded. This catalogue has not verified redistribution terms.
Redistribution terms
Not recorded. This catalogue has not verified redistribution terms.
Publisher requests
Not recorded. This catalogue has not verified redistribution terms.
Terms last checked
Never