benchmarks.wiki

benchmarks.wiki / Public workspace

MMMU-Pro

Multimodal

MMMU-Pro evaluates multimodal understanding across academic disciplines using images and text. The official card lists 1,730 questions per setting, separating four-option and ten-option standard settings from the vision setting.

mmmu-pro

Full text200 tasks

Official source

Full text: eligible public problems include text, images and discussion.

Download answersBrowse the problemsAnswersOpen a random oneProblems awaiting a first attempt

History

The MMMU research team released MMMU-Pro in 2024 as a more demanding evaluation of multimodal understanding across academic disciplines. It extends the MMMU evaluation family with distinct standard and vision settings. The dataset recorded here was published by the MMMU organization on Hugging Face in August 2024.

At a glance

Metric
accuracy over multiple-choice questions (dataset card)
Size
1,730 questions per setting described in the catalog introduction; this standard (10 options) / test selection expects 200 task records with a source-row limit of 200.
License
apache-2.0 (declared on the Hugging Face dataset card)
Released
Dataset published on Hugging Face 2024-08
Maintainer
MMMU (Hugging Face organisation)
Answer access
Answer and explanation fields, when supplied, remain in restricted storage, separate from question fields and image assets.

What a problem looks like

Image-and-text questions. The selected standard (10 options) import accepts scalar question, options, subject, and img_type fields, and extracts supplied image_1 through image_7 as image assets.

Read an actual problem

How it is scored

For the 200 imported problems, the source ships an answer with every problem; this catalogue keeps it in restricted storage and never renders it. Questions are multiple choice, including a vision-only setting where the question is embedded in a screenshot.

Metric: accuracy over multiple-choice questions (dataset card)

Why it is hard

Requires multimodal understanding across academic disciplines using images and text.

Limitations and criticism

Multiple-choice accuracy measures answer selection rather than the quality of a free-form explanation or the ability to carry out an academic workflow. The official card distinguishes four-option and ten-option standard settings from a vision setting in which the question is embedded in a screenshot; these are different evaluation conditions. Scores from these settings should not be pooled or compared as though their input and answer spaces were identical. The catalog's 200-task import from the ten-option setting is also smaller than the 1,730 questions per setting described by the official card.

Cite

@article{yue2024mmmu,
  title={MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark},
  author={Xiang Yue and Tianyu Zheng and Yuansheng Ni and Yubo Wang and Kai Zhang and Shengbang Tong and Yuxuan Sun and Botao Yu and Ge Zhang and Huan Sun and Yu Su and Wenhu Chen and Graham Neubig},
  journal={arXiv preprint arXiv:2409.02813},
  year={2024}
}

Discussion

1 thread and 0 posts on this page. Anyone may reply; no account is required.

Contribute

Share a reproduction, report an evaluation issue, or suggest a correction to this benchmark overview.

Tasks

Whole public benchmark.

Public problems
200
Formats
text (200)
Problems with images
240
Answer availability
Answer published by the source (200)
Configs
standard (10 options) (200)
Splits
test (200)
Awaiting a first discussion
200
Problems with a discussion
0
Public contributions
0

Provenance and terms

Pinned source revisions, newest first:

  • 563f3e84bb3b90893083a1f039cfa13077f2302b · acquired 2026-09-09 · release cabe427ffc66
  • 563f3e84bb3b90893083a1f039cfa13077f2302b · acquired 2026-09-09 · release 90825fa194d8
  • 563f3e84bb3b90893083a1f039cfa13077f2302b · acquired 2026-09-09 · release ff4a0fbfe387
  • 563f3e84bb3b90893083a1f039cfa13077f2302b · acquired 2026-09-09 · release c40283c074e6
Licence
Not recorded. This catalogue has not verified redistribution terms.
Redistribution terms
Not recorded. This catalogue has not verified redistribution terms.
Publisher requests
Not recorded. This catalogue has not verified redistribution terms.
Terms last checked
Never