benchmarks.wiki

benchmarks.wiki / Public workspace

SimpleQA

Knowledge / Search

SimpleQA collects short fact-seeking questions that have exactly one indisputable answer, measuring both how often a model is right and how often it declines. The format is deliberately narrow so that grading is not itself contestable.

simpleqa

Full text4,326 tasks

Official source

Full text: eligible public problems include text, images and discussion.

Download answersBrowse the problemsAnswersOpen a random oneProblems awaiting a first attempt

History

OpenAI introduced SimpleQA in 2024 to measure factual accuracy on short questions with narrowly specified answers. It was designed to make factual grading more tractable than evaluation of long responses containing many separate claims. Its evaluation distinguishes correct answers, incorrect answers, and cases where a model does not attempt an answer.

How it was built

SimpleQA was assembled from human-written fact-seeking questions with reference answers and supporting sources. The questions were designed to have a single verifiable answer rather than require an open-ended explanation. The source contains 4,326 questions; this catalogue imports 200 test records and keeps both reference answers and documenting URLs outside the prompt.

At a glance

Metric
correct, incorrect and not-attempted rates, and the F-score that combines them
Size
4,326 problems imported from the default / test selection (the full split).
License
mit (declared on the Hugging Face dataset card)
Released
Dataset published on Hugging Face 2024-11
Maintainer
basicv8vc (source organisation)
Answer access
Columns outside the prompt allowlist, including any answer or reference solution, are separated into restricted storage and never rendered.

What a problem looks like

One short question. The reference answer and the URLs that document it are not part of the prompt.

Read an actual problem

How it is scored

The source ships a reference answer with every question; this catalogue keeps it in restricted storage and never renders it.

Metric: correct, incorrect and not-attempted rates, and the F-score that combines them

Why it is hard

The questions are short but the facts are obscure, and a model that guesses scores worse than one that abstains. It measures calibration as much as knowledge.

Limitations and criticism

Short factual questions do not measure whether a long response remains accurate across many claims, or whether a system can synthesize a useful explanation. The intended answerability of the questions does not eliminate possible ambiguity or changes in the underlying facts. Access to search or other retrieval tools changes the capability being evaluated, so tool-assisted and closed-book results need separate interpretation. Correct, incorrect, and not-attempted rates also carry information that a single aggregate score can conceal.

Cite

@article{wei2024simpleqa,
  title={Measuring short-form factuality in large language models},
  author={Jason Wei and Nguyen Karina and Hyung Won Chung and Yunxin Joy Jiao and Spencer Papay and Amelia Glaese and John Schulman and William Fedus},
  journal={arXiv preprint arXiv:2411.04368},
  year={2024}
}

Discussion

1 thread and 0 posts on this page. Anyone may reply; no account is required.

Contribute

Share a reproduction, report an evaluation issue, or suggest a correction to this benchmark overview.

Tasks

Whole public benchmark.

Public problems
4,326
Formats
text (4,326)
Problems with images
0
Answer availability
Answer published by the source (4,325)
Configs
default (4,326)
Splits
test (4,326)
Awaiting a first discussion
4,326
Problems with a discussion
0
Public contributions
0

Provenance and terms

Pinned source revisions, newest first:

  • e319282ab125c3dbd0c7fd00be2e4dd54e7e8f94 · acquired 2026-09-12 · release 2f00e67ed0eb
  • e319282ab125c3dbd0c7fd00be2e4dd54e7e8f94 · acquired 2026-09-12 · release 2f3ac1c4c266
Licence
Not recorded. This catalogue has not verified redistribution terms.
Redistribution terms
Not recorded. This catalogue has not verified redistribution terms.
Publisher requests
Not recorded. This catalogue has not verified redistribution terms.
Terms last checked
Never