benchmarks.wiki

benchmarks.wiki / Public workspace

AIME 2024

Mathematics

The 30 problems of the 2024 American Invitational Mathematics Examination (AIME I and II). Every answer is an integer from 0 to 999, with no options and no partial credit. It is a frequent reference point for reasoning models.

aime-2024

Full text30 tasks

Official source

Full text: eligible public problems include text, images and discussion.

Download answersBrowse the problemsAnswersOpen a random oneProblems awaiting a first attempt

History

The 2024 American Invitational Mathematics Examination was a mathematics competition administered by the Mathematical Association of America, rather than a test originally designed for language models. Its short numerical answers and demanding solutions made it useful for evaluating mathematical reasoning. Maxwell-Jia published the dataset version recorded here on Hugging Face in December 2024.

How it was built

The dataset collects all 30 problems from the 2024 AIME I and AIME II examinations. These are human-authored competition problems, each requiring an integer answer from 0 to 999 rather than a choice among supplied options. The source includes worked solutions and final answers, which are excluded from the evaluation prompt.

At a glance

Size
All 30 problems imported from the default / train selection, with a source-row limit of 200.
License
mit (declared on the Hugging Face dataset card)
Released
Dataset published on Hugging Face 2024-12
Maintainer
Maxwell-Jia (source organisation)
Answer access
Columns outside the prompt allowlist, including any answer or reference solution, are separated into restricted storage and never rendered.
Highest recorded
39.2 Pass@1 — DeepSeek V3 (vendor, self-reported)

What a problem looks like

One competition problem statement. The official solution and the integer answer are not part of the prompt.

Read an actual problem

How it is scored

The source ships the worked solution and the integer answer; this catalogue keeps both in restricted storage and never renders them.

Reported scores

Evaluation conditions differ between rows, so this is not a ranking. Rows marked vendor are reports by the model’s own organization.

ModelScoreMetricSetupReported bySource
DeepSeek V3DeepSeek39.2Pass@1Chat model; 8K output limit; multiple runs with varying temperatures for benchmarks under 1000 samples.vendor (self-reported)Chat Model, Standard Benchmarks, AIME 2024 (Pass@1), final column

Why it is hard

Each problem takes a multi-step construction that has to be carried out exactly. A single arithmetic slip produces a wrong integer, and thirty problems make each one worth three points of the score.

Limitations and criticism

Thirty problems provide a small sample of mathematical ability: one additional correct answer changes single-run accuracy by about 3.3 percentage points. Final-answer grading gives no partial credit and does not establish that the accompanying reasoning is valid. The public availability of the problems and solutions creates a potential training-contamination risk, while sampling, output limits and repeated attempts can change reported performance.

How much room is left

The vendor report recorded here gives DeepSeek V3 a pass@1 score of 39.2 percent, using an 8K output limit and repeated runs with varying temperatures for small benchmarks. That result leaves substantial headroom in the reported setup, but it does not establish the present best performance. Fractional scores from repeated sampling should not be interpreted as the number of problems solved in a single 30-question attempt.

Discussion

1 thread and 0 posts on this page. Anyone may reply; no account is required.

Contribute

Share a reproduction, report an evaluation issue, or suggest a correction to this benchmark overview.

Tasks

Whole public benchmark.

Public problems
30
Formats
math (30)
Problems with images
0
Answer availability
Answer published by the source (30)
Configs
default (30)
Splits
train (30)
Awaiting a first discussion
30
Problems with a discussion
0
Public contributions
0

Provenance and terms

Pinned source revisions, newest first:

  • 8d88b2876a82a080e2f172cc9b25d0d9d2cb4792 · acquired 2026-09-12 · release 31c7e41d93d9
  • 8d88b2876a82a080e2f172cc9b25d0d9d2cb4792 · acquired 2026-09-12 · release a0ba82c12c70
  • 8d88b2876a82a080e2f172cc9b25d0d9d2cb4792 · acquired 2026-09-12 · release 9986a9443c45
Licence
Not recorded. This catalogue has not verified redistribution terms.
Redistribution terms
Not recorded. This catalogue has not verified redistribution terms.
Publisher requests
Not recorded. This catalogue has not verified redistribution terms.
Terms last checked
Never