benchmarks.wiki

benchmarks.wiki / Public workspace

ARC-Challenge

Reasoning / Science

ARC-Challenge is the subset of grade-school science questions that both a retrieval baseline and a word co-occurrence baseline answered incorrectly. It tests reasoning with knowledge rather than recall alone.

arc-challenge

Full text1,172 tasks

Official source

Full text: eligible public problems include text, images and discussion.

Download answersBrowse the problemsAnswersOpen a random oneProblems awaiting a first attempt

History

The Allen Institute for Artificial Intelligence introduced the AI2 Reasoning Challenge in 2018 to test question answering that required more than straightforward retrieval. ARC-Challenge is its difficult subset of grade-school science questions. The March 2022 date recorded for the Hugging Face distribution is an upload date rather than the original introduction.

How it was built

ARC collects multiple-choice questions from existing science examinations. The Challenge subset retains questions that both a retrieval baseline and a word co-occurrence baseline answered incorrectly. Questions have three to five answer options, and the source test split contains 1,172 items; the catalogue selection contains 200 of these.

At a glance

Size
1,172 problems imported from the ARC-Challenge / test selection (the full test split).
License
cc-by-sa-4.0 (declared on the Hugging Face dataset card)
Released
Dataset published on Hugging Face 2022-03
Maintainer
allenai (source organisation)
Answer access
Columns outside the prompt allowlist, including any answer or reference solution, are separated into restricted storage and never rendered.
Highest recorded
95.3 Acc. — DeepSeek-V3 (vendor, self-reported)

What a problem looks like

A science question with three to five options. The options are imported as a list; the answer key is not.

Read an actual problem

How it is scored

The source ships an answer key with every problem; this catalogue keeps it in restricted storage and never renders it.

Reported scores

Evaluation conditions differ between rows, so this is not a ranking. Rows marked vendor are reports by the model’s own organization.

ModelScoreMetricSetupReported bySource
DeepSeek-V3DeepSeek95.3Acc.Base model; 25-shot.vendor (self-reported)Base Model, Standard Benchmarks, ARC-Challenge (Acc.), final column

Why it is hard

The set was built by discarding everything the simple baselines could already do, so what remains needs a chain of reasoning over the science involved rather than a lookup.

Limitations and criticism

Failure by the original baselines is a selection criterion, not proof that every retained question requires multi-step reasoning. Multiple-choice accuracy combines scientific knowledge, reasoning and exploitation of answer cues, without directly assessing explanations. Public availability creates a risk of training exposure, and prompting or option-scoring conventions can affect comparisons.

How much room is left

The recorded vendor report gives DeepSeek-V3 base 95.3 percent accuracy with 25 examples in the prompt. That result leaves little headroom in that particular setup, limiting the room for aggregate accuracy to distinguish models. It does not establish equivalent performance under other prompting protocols or on science questions outside the fixed test set.

Cite

@article{clark2018arcchallenge,
  title={Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge},
  author={Peter Clark and Isaac Cowhey and Oren Etzioni and Tushar Khot and Ashish Sabharwal and Carissa Schoenick and Oyvind Tafjord},
  journal={arXiv preprint arXiv:1803.05457},
  year={2018}
}

Discussion

1 thread and 0 posts on this page. Anyone may reply; no account is required.

Contribute

Share a reproduction, report an evaluation issue, or suggest a correction to this benchmark overview.

Tasks

Whole public benchmark.

Public problems
1,172
Formats
text (1,172)
Problems with images
0
Answer availability
Answer published by the source (1,172)
Configs
ARC-Challenge (1,172)
Splits
test (1,172)
Awaiting a first discussion
1,172
Problems with a discussion
0
Public contributions
0

Provenance and terms

Pinned source revisions, newest first:

  • 210d026faf9955653af8916fad021475a3f00453 · acquired 2026-09-12 · release 64068a43aa57
  • 210d026faf9955653af8916fad021475a3f00453 · acquired 2026-09-12 · release 78d02a92bc40
  • 210d026faf9955653af8916fad021475a3f00453 · acquired 2026-09-12 · release a14a6faab45b
Licence
Not recorded. This catalogue has not verified redistribution terms.
Redistribution terms
Not recorded. This catalogue has not verified redistribution terms.
Publisher requests
Not recorded. This catalogue has not verified redistribution terms.
Terms last checked
Never