benchmarks.wiki

benchmarks.wiki / Public workspace

MMMU

Multimodal

MMMU collects 11,500 college-level questions across six disciplines and thirty subjects, each carrying figures, diagrams or images that have to be read as well as reasoned over. MMMU-Pro is its tightened revision.

mmmu

Official source

Download answersVisit the official benchmarkWhere the answers areStart a benchmark discussion

History

The MMMU research team introduced the benchmark in 2023 to assess multimodal understanding and reasoning at the level of college subject knowledge. Its source dataset was published under the MMMU organisation on Hugging Face in November 2023. The benchmark aimed to extend evaluation beyond everyday image recognition to questions requiring specialist knowledge and interpretation of academic visual material.

How it was built

MMMU collects 11,500 college-level questions across six disciplines and 30 subjects. Questions were gathered from educational materials and combine text with visual content such as diagrams, charts and other images. Its subject coverage makes the evaluation a mixture of domain knowledge and interpretation of visual evidence, rather than a collection of interchangeable image-recognition tasks.

At a glance

Size
11,500 questions in the source; none imported.
License
apache-2.0 (declared on the Hugging Face dataset card)
Released
Dataset published on Hugging Face 2023-11
Maintainer
MMMU (source organisation)
Answer access
No problem text is imported, so there is no answer storage for this entry. The source distributes the answers under its own terms.

What a problem looks like

A question with several images or diagrams and a set of options.

How it is scored

The source ships the answer; nothing is imported here.

Why it is hard

The figures carry information the text does not repeat, so a model that reads only the question is answering a different, easier question.

Limitations and criticism

An aggregate MMMU score combines specialist knowledge, visual interpretation and answer selection, which makes it difficult to attribute a failure to any one capability. Performance can also vary substantially between subjects, so the overall number can obscure uneven coverage. Multiple-choice items allow correct selections without demonstrating a complete explanation. MMMU-Pro is a revised benchmark and its results should not be treated as scores on the original MMMU set.

Cite

@article{yue2023mmmu,
  title={MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI},
  author={Xiang Yue and Yuansheng Ni and Kai Zhang and Tianyu Zheng and Ruoqi Liu and Ge Zhang and Samuel Stevens and Dongfu Jiang and Weiming Ren and Yuxuan Sun and Cong Wei and Botao Yu and Ruibin Yuan and Renliang Sun and Ming Yin and Boyuan Zheng and Zhenzhu Yang and Yibo Liu and Wenhao Huang and Huan Sun and Yu Su and Wenhu Chen},
  journal={arXiv preprint arXiv:2311.16502},
  year={2023}
}

Discussion

1 thread and 0 posts on this page. Anyone may reply; no account is required.

Contribute

Share a reproduction, report an evaluation issue, or suggest a correction to this benchmark overview.

Public problem summary

Catalogue metadata only; public problem statistics are not available.

Public problems
—
Formats
—
Problems with images
—
Answer availability
—
Configs
—
Splits
—
Awaiting a first discussion
—
Problems with a discussion
—
Public contributions
—