benchmarks.wiki

benchmarks.wiki / Public workspace

OlympiadBench

Mathematics / Science / Multimodal

OlympiadBench evaluates scientific reasoning on Olympiad-level mathematics and physics problems. Its official description lists 8,476 English and Chinese problems with separate text-only and multimodal settings.

olympiadbench

Full text200 tasks

Official source

Full text: eligible public problems include text, images and discussion.

Download answersBrowse the problemsAnswersOpen a random oneProblems awaiting a first attempt

History

The OlympiadBench research team introduced the benchmark in 2024 to evaluate scientific reasoning on difficult mathematics and physics problems. Its official repository is hosted by OpenBMB, and the Hothan dataset distribution on Hugging Face was published in July 2024. The benchmark includes English and Chinese material and both text-only and multimodal settings.

How it was built

OlympiadBench draws on existing Olympiad and examination problems in mathematics and physics. The official description lists 8,476 problems across its English and Chinese settings, including tasks with visual content. The catalog's selection contains 200 records from an English text-only mathematics competition setting, rather than a sample covering every setting. Records distinguish properties such as answer type, units and whether multiple answers are expected.

At a glance

Size
8,476 English and Chinese problems described across settings in the catalog introduction; this OE_TO_maths_en_COMP / train selection expects 200 task records with a source-row limit of 200.
License
apache-2.0 (declared on the Hugging Face dataset card)
Released
Dataset published on Hugging Face 2024-07
Maintainer
Hothan (Hugging Face organisation)
Answer access
Fields outside the question allowlist, including answers and solutions when supplied, remain in restricted storage; the converter alone does not establish whether usable answers are available.

What a problem looks like

The selected OE_TO_maths_en_COMP setting contains text-only mathematics questions. The import accepts question, language, subject, answer_type, question_type, unit, and is_multiple_answer.

Read an actual problem

How it is scored

For the 200 imported problems, the source ships an answer with every problem; this catalogue keeps it in restricted storage and never renders it.

Why it is hard

Requires scientific reasoning on Olympiad-level mathematics and physics problems.

Limitations and criticism

An OlympiadBench score needs to identify the evaluated language, subject and text-only or multimodal setting. Performance on English text-only mathematics cannot be treated as evidence of equal ability on Chinese problems or visual physics tasks. Because the benchmark draws on existing examination material, training-data overlap is a potential concern rather than something a high score can rule out. The tasks assess solving posed problems, not open-ended scientific investigation.

Cite

@article{he2024olympiadbench,
  title={Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems},
  author={He, Chaoqun and Luo, Renjie and Bai, Yuzhuo and Hu, Shengding and Thai, Zhen Leng and Shen, Junhao and Hu, Jinyi and Han, Xu and Huang, Yujie and Zhang, Yuxiang and others},
  journal={arXiv preprint arXiv:2402.14008},
  year={2024}
}

Discussion

1 thread and 0 posts on this page. Anyone may reply; no account is required.

Contribute

Share a reproduction, report an evaluation issue, or suggest a correction to this benchmark overview.

Tasks

Whole public benchmark.

Public problems
200
Formats
text (200)
Problems with images
0
Answer availability
Answer published by the source (200)
Configs
OE_TO_maths_en_COMP (200)
Splits
train (200)
Awaiting a first discussion
200
Problems with a discussion
0
Public contributions
0

Provenance and terms

Pinned source revisions, newest first:

  • 91184b52131e7fc9455fef848035173aea8cc01a · acquired 2026-09-09 · release 54350c3251ca
  • 91184b52131e7fc9455fef848035173aea8cc01a · acquired 2026-09-09 · release e6b5c6cafc50
  • 91184b52131e7fc9455fef848035173aea8cc01a · acquired 2026-09-09 · release 479001f7f1d0
  • 91184b52131e7fc9455fef848035173aea8cc01a · acquired 2026-09-09 · release f67d450557c0
Licence
Not recorded. This catalogue has not verified redistribution terms.
Redistribution terms
Not recorded. This catalogue has not verified redistribution terms.
Publisher requests
Not recorded. This catalogue has not verified redistribution terms.
Terms last checked
Never