benchmarks.wiki

benchmarks.wiki / Public workspace

GSM8K

Mathematics

GSM8K is a set of grade-school word problems that each need several arithmetic steps. It remains a standard reference point for multi-step reasoning.

gsm8k

Full text1,319 tasks

Official source

Full text: eligible public problems include text, images and discussion.

Download answersBrowse the problemsAnswersOpen a random oneProblems awaiting a first attempt

History

OpenAI introduced GSM8K in 2021 as a benchmark of multistep mathematical reasoning expressed in ordinary language. The work examined whether models could produce and assess solutions to grade-school word problems, where individual calculations are elementary but must be combined correctly. The April 2022 Hugging Face publication recorded for this source is the date of the hosted source, not the benchmark's original introduction.

How it was built

GSM8K consists of 8,792 human-written grade-school mathematics word problems with worked solutions. The standard split contains 7,473 training problems and 1,319 test problems. Problems require several dependent arithmetic steps, and evaluation typically extracts the final numerical answer from the generated solution. This catalogue imports 200 problems from the main test split, rather than the complete test set.

At a glance

Size
1,319 problems imported from the main / test selection (the full test split).
License
mit (declared on the source)
Released
Dataset published on Hugging Face 2022-04
Maintainer
openai (source organisation)
Answer access
Columns outside the prompt allowlist, including any answer or reference solution, are separated into restricted storage and never rendered.
Highest recorded
89.3 EM — DeepSeek-V3 (vendor, self-reported)

What a problem looks like

A short word problem in the question field. The worked solution and final answer are kept out of the prompt.

Read an actual problem

How it is scored

The source ships an answer with every problem; this catalogue keeps it in restricted storage and never renders it.

Reported scores

Evaluation conditions differ between rows, so this is not a ranking. Rows marked vendor are reports by the model’s own organization.

ModelScoreMetricSetupReported bySource
DeepSeek-V3DeepSeek89.3EMBase model; 8-shot.vendor (self-reported)Base Model, Standard Benchmarks, GSM8K (EM), final column

Why it is hard

Each problem is elementary in isolation; the difficulty is carrying several dependent steps without arithmetic slips.

Limitations and criticism

GSM8K covers a restricted range of short arithmetic word problems and does not measure advanced mathematics, proof construction or visual reasoning. Final-answer exact match does not verify each step of a solution, and answer extraction can affect the measured score. Because both the problems and worked solutions are publicly distributed, benchmark accuracy alone cannot distinguish general reasoning from prior exposure. Results also depend on whether models use tools, multiple samples or selection among candidate solutions.

How much room is left

The vendor report recorded here records 89.3 exact match for the DeepSeek-V3 base model in an eight-shot setup. That result demonstrates strong performance while leaving errors in the reported configuration; it does not establish the ceiling reached by other systems. The remaining gap should be interpreted alongside the evaluation setup and the benchmark's narrow arithmetic coverage, rather than as a general measure of mathematical capability.

Cite

@article{cobbe2021gsm8k,
  title={Training Verifiers to Solve Math Word Problems},
  author={Cobbe, Karl and Kosaraju, Vineet and Bavarian, Mohammad and Chen, Mark and Jun, Heewoo and Kaiser, Lukasz and Plappert, Matthias and Tworek, Jerry and Hilton, Jacob and Nakano, Reiichiro and Hesse, Christopher and Schulman, John},
  journal={arXiv preprint arXiv:2110.14168},
  year={2021}
}

Discussion

1 thread and 0 posts on this page. Anyone may reply; no account is required.

Contribute

Share a reproduction, report an evaluation issue, or suggest a correction to this benchmark overview.

Tasks

Whole public benchmark.

Public problems
1,319
Formats
text (1,319)
Problems with images
0
Answer availability
Answer published by the source (1,319)
Configs
main (1,319)
Splits
test (1,319)
Awaiting a first discussion
1,319
Problems with a discussion
0
Public contributions
0

Provenance and terms

Pinned source revisions, newest first:

  • 740312add88f781978c0658806c59bc2815b9866 · acquired 2026-09-11 · release a81f6d44d318
  • 740312add88f781978c0658806c59bc2815b9866 · acquired 2026-09-11 · release c754bdfb5f7c
  • 740312add88f781978c0658806c59bc2815b9866 · acquired 2026-09-11 · release a8f963ca5e1e
Licence
Not recorded. This catalogue has not verified redistribution terms.
Redistribution terms
Not recorded. This catalogue has not verified redistribution terms.
Publisher requests
Not recorded. This catalogue has not verified redistribution terms.
Terms last checked
Never