benchmarks.wiki

benchmarks.wiki / Public workspace

Humanity's Last Exam

Reasoning / Multimodal

Humanity's Last Exam evaluates AI knowledge and reasoning across academic disciplines. It includes text and image questions; this catalog provides an introduction and official links in accordance with the publisher's request.

hle

Official source

Download answersVisit the official benchmarkWhere the answers areStart a benchmark discussion

History

The Center for AI Safety and Scale AI introduced Humanity's Last Exam in 2025 to provide a difficult test of academic knowledge and reasoning as existing benchmarks became less discriminating. The dataset was published on Hugging Face in January 2025. Its questions span academic disciplines and include both text-only and image-based material.

How it was built

Humanity's Last Exam was assembled from questions contributed by subject-matter experts across academic disciplines. Candidate questions underwent expert review and screening against contemporary models to select challenging items. The resulting evaluation includes text and image questions, so success can require both specialist knowledge and interpretation of visual material.

At a glance

License
mit (declared on the Hugging Face dataset card)
Released
Dataset published on Hugging Face 2025-01
Maintainer
cais (Hugging Face organisation)
Answer access
This catalog has not downloaded questions or answers; its acquisition record notes gated access and a publisher request against republication.
Highest recorded
38.3 Accuracy (%) — Gemini 3 Pro (leaderboard)

What a problem looks like

Text and image questions across academic disciplines.

Reported scores

Evaluation conditions differ between rows, so this is not a ranking. Rows marked vendor are reports by the model’s own organization.

ModelScoreMetricSetupReported bySource
Gemini 3 Pro38.3Accuracy (%)Dataset updated April 3, 2025; judge model o3-mini.leaderboardQuantitative Results, Accuracy column
GPT-525.3Accuracy (%)Dataset updated April 3, 2025; judge model o3-mini.leaderboardQuantitative Results, Accuracy column
Grok 424.5Accuracy (%)Dataset updated April 3, 2025; judge model o3-mini.leaderboardQuantitative Results, Accuracy column

Why it is hard

Requires knowledge and reasoning across academic disciplines.

Limitations and criticism

Performance on difficult academic questions does not directly measure sustained research, experimental design or successful action in an external environment. Screening questions for difficulty against existing models also makes the collection a deliberately selected challenge set rather than a representative sample of everyday knowledge tasks. The recorded leaderboard uses an updated dataset and an o3-mini judge, so dataset revision and answer-grading procedure are part of the evaluation conditions.

How much room is left

The recorded official leaderboard reports 38.3% accuracy for Gemini 3 Pro, 25.3% for GPT-5 and 24.5% for Grok 4. These entries use the dataset updated on April 3, 2025 and an o3-mini judge. The reported results leave substantial headroom under that setup, while comparisons with other reports require matching the dataset version and grading conditions.

Discussion

1 thread and 0 posts on this page. Anyone may reply; no account is required.

Contribute

Share a reproduction, report an evaluation issue, or suggest a correction to this benchmark overview.

Public problem summary

Catalogue metadata only; public problem statistics are not available.

Public problems
—
Formats
—
Problems with images
—
Answer availability
—
Configs
—
Splits
—
Awaiting a first discussion
—
Problems with a discussion
—
Public contributions
—