benchmarks.wiki / Public workspace
Compare the whole catalogue · Browse model reports
Choose a field to explore public problems and discussions.
Fields and counts refer to this catalog page.
Reasoning
arc-agi-2
ARC-AGI-2 evaluates the ability to infer unfamiliar rules from a few examples and apply them to new grids. Training and evaluation are separate; this import covers the 120 public evaluation tasks.
Repository: Source repository
Reasoning / Agents
arc-agi-3
ARC-AGI-3 is an interactive benchmark for exploring unfamiliar game environments, learning their rules, and planning actions. The official toolkit provides an environment interface, with game versions and trial records t…
Reasoning / Science
arc-challenge
ARC-Challenge is the subset of grade-school science questions that both a retrieval baseline and a word co-occurrence baseline answered incorrectly. It tests reasoning with knowledge rather than recall alone.
Paper: arXiv:1803.05457
Reasoning / Science
arc-easy
ARC-Easy is the subset of grade-school science questions that a retrieval baseline or a word co-occurrence baseline answered correctly. It is the easier counterpart of ARC-Challenge and serves as the baseline measure of …
Paper: arXiv:1803.05457
Reasoning
bbeh
BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categorie…
Repository: Source repository
Reasoning
bbh
BIG-Bench Hard is the set of 23 BIG-Bench tasks on which models of the time scored below the average human rater. It is best known for showing how much chain-of-thought prompting changes the result.
Repository: Official repository
Reasoning
boolq
BoolQ pairs naturally occurring yes/no questions from search queries with a passage that decides them. The format is simple, but the inference it needs is consistently heavier than it looks.
Repository: Official repository
Code / Reasoning
cruxeval
CRUXEval takes 800 short Python functions and asks either for the output given an input (CRUXEval-O) or for an input that produces a given output (CRUXEval-I). It measures the ability to execute code mentally rather than…
Paper: arXiv:2401.03065
Reasoning
drop
DROP is a reading-comprehension benchmark whose questions require discrete operations over the paragraph — addition, subtraction, comparison, counting, sorting. It is built so that extracting a span cannot answer them.
Paper: arXiv:1903.00161
Reasoning
hellaswag
HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.
Paper: arXiv:1905.07830
Reasoning / Multimodal
hle
Humanity's Last Exam evaluates AI knowledge and reasoning across academic disciplines. It includes text and image questions; this catalog provides an introduction and official links in accordance with the publisher's req…
Reasoning / Instruction following
ifeval
IFEval measures whether a model follows verifiable instructions -- word counts, formats, forbidden words -- rather than whether an answer is good. Every constraint is checked programmatically.
Paper: arXiv:2311.07911
Reasoning / Long context
longbench-v2
LongBench v2 evaluates deep understanding and reasoning over long contexts through multiple-choice questions. Its official description lists 503 questions spanning tasks such as single-document and multi-document QA and …
Paper: arXiv:2412.15204
Reasoning / Knowledge
mmlu
MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.
Repository: Official repository
Reasoning
mmlu-pro
MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.
Paper: arXiv:2406.01574
Reasoning
musr
MuSR poses multistep soft reasoning problems over algorithmically generated narratives. It has three domains — murder mysteries, object placements and team allocation — and this catalogue imports all three domains, 756 p…
Paper: arXiv:2310.16049