benchmarks.wiki

benchmarks.wiki / Public workspace

MGSM

Mathematics / Multilingual

MGSM is a human translation of 250 GSM8K problems into eleven languages, measuring whether reasoning survives the change of language. The gap against English is the multilingual signal.

mgsm

Full text2,750 tasks

Official source

Full text: eligible public problems include text, images and discussion.

Download answersBrowse the problemsAnswersOpen a random oneProblems awaiting a first attempt

History

Freda Shi and collaborators at Google Research introduced MGSM in 2022 to investigate whether chain-of-thought reasoning could work across languages. They adapted grade-school mathematics problems from GSM8K so that the same underlying reasoning task could be compared across language settings. The Hugging Face source listed in the catalogue was published in May 2023, after the original research.

How it was built

MGSM uses human translations of 250 GSM8K problems and covers eleven languages, including English. The parallel problems preserve the underlying arithmetic while changing the language in which the task is presented. The source supplies answers, numeric answers, and worked equations; this catalogue imports 200 records from the English test selection and separates those reference fields from the prompt.

At a glance

Size
2,750 problems imported from the original TSVs: the same 250 grade-school problems in each of the eleven languages, kept as per-language pages.
License
cc-by-4.0 (declared in the original repository mgsm/LICENSE)
Released
Dataset published on Hugging Face 2023-05
Maintainer
juletxara (source organisation)
Answer access
Columns outside the prompt allowlist, including any answer or reference solution, are separated into restricted storage and never rendered.
Highest recorded
79.8 EM — DeepSeek-V3 (vendor, self-reported)

What a problem looks like

One grade-school word problem in the selected language. The answer, the numeric answer and the worked equation are not part of the prompt.

Read an actual problem

How it is scored

The source ships the answer, the numeric answer and the equation; this catalogue keeps all three in restricted storage and never renders them.

Reported scores

Evaluation conditions differ between rows, so this is not a ranking. Rows marked vendor are reports by the model’s own organization.

ModelScoreMetricSetupReported bySource
DeepSeek-V3DeepSeek79.8EMBase model; 8-shot.vendor (self-reported)Base Model, Standard Benchmarks, MGSM (EM), final column

Why it is hard

The arithmetic is elementary, so anything lost against the English score is lost in the language, not the mathematics — including in languages with little presence in training data.

Limitations and criticism

The arithmetic is elementary, so MGSM does not measure advanced mathematical reasoning or broad multilingual competence. Parallel translations support controlled comparisons, but wording and translation choices can affect difficulty as well as model language ability. The English-only catalogue selection cannot by itself measure a multilingual performance gap. Shared problems from the public GSM8K source also create a potential path for training-data overlap.

How much room is left

The catalogue records a vendor-reported exact-match score of 79.8 for the DeepSeek-V3 base model with eight-shot prompting. That result remains below the ceiling under the reported setup, but a single recorded score cannot establish the best available performance. Language-level breakdowns and matched prompting are needed to assess whether remaining errors concentrate in particular languages.

Cite

@article{shi2022mgsm,
  title={Language Models are Multilingual Chain-of-Thought Reasoners},
  author={Freda Shi and Mirac Suzgun and Markus Freitag and Xuezhi Wang and Suraj Srivats and Soroush Vosoughi and Hyung Won Chung and Yi Tay and Sebastian Ruder and Denny Zhou and Dipanjan Das and Jason Wei},
  journal={arXiv preprint arXiv:2210.03057},
  year={2022}
}

Discussion

1 thread and 0 posts on this page. Anyone may reply; no account is required.

Contribute

Share a reproduction, report an evaluation issue, or suggest a correction to this benchmark overview.

Tasks

Whole public benchmark.

Public problems
2,750
Formats
math (2,750)
Problems with images
0
Answer availability
Answer published by the source (2,750)
Configs
bn (250) · de (250) · en (250) · es (250) · fr (250) · ja (250) · ru (250) · sw (250) · te (250) · th (250) · zh (250)
Splits
test (2,750)
Awaiting a first discussion
2,750
Problems with a discussion
0
Public contributions
0

Provenance and terms

Pinned source revisions, newest first:

  • 452a21ad3dae5668c06ceeac21ff073e1e40f9be · acquired 2026-09-23 · release 1017bb4fe10b
  • b2f13d426afe3be8d69a7e739b36724db8b66bbc · acquired 2026-09-12 · release d28706c95f1e
  • b2f13d426afe3be8d69a7e739b36724db8b66bbc · acquired 2026-09-12 · release a7c8fef5f04f
Licence
Not recorded. This catalogue has not verified redistribution terms.
Redistribution terms
Not recorded. This catalogue has not verified redistribution terms.
Publisher requests
Not recorded. This catalogue has not verified redistribution terms.
Terms last checked
Never