benchmarks.wiki / Public workspace
Compare the whole catalogue · Browse model reports
Choose a field to explore public problems and discussions.
Fields and counts refer to this catalog page.
Safety / Agents
agentharm
AgentHarm measures how far a tool-using agent will go in carrying out a harmful request. It pairs 110 harmful behaviours with 110 matched benign ones, so over-refusal is measured alongside compliance.
Paper: arXiv:2410.09024
Mathematics
aime-2024
The 30 problems of the 2024 American Invitational Mathematics Examination (AIME I and II). Every answer is an integer from 0 to 999, with no options and no partial credit. It is a frequent reference point for reasoning m…
Mathematics
aime-2025
The 15 problems of AIME I 2025. As one of the newest competition sets available, it leaves the least room for training-data contamination, and reading it beside the 2024 set makes that effect visible.
Mathematics
amc
83 AMC 12 problems assembled by Project Numina as a validation set for the AIMO competition. They sit a step below AIME in difficulty, which makes them a useful lower rung on the same scale.
Reasoning
arc-agi-2
ARC-AGI-2 evaluates the ability to infer unfamiliar rules from a few examples and apply them to new grids. Training and evaluation are separate; this import covers the 120 public evaluation tasks.
Repository: Source repository
Reasoning / Agents
arc-agi-3
ARC-AGI-3 is an interactive benchmark for exploring unfamiliar game environments, learning their rules, and planning actions. The official toolkit provides an environment interface, with game versions and trial records t…
Reasoning / Science
arc-challenge
ARC-Challenge is the subset of grade-school science questions that both a retrieval baseline and a word co-occurrence baseline answered incorrectly. It tests reasoning with knowledge rather than recall alone.
Paper: arXiv:1803.05457
Reasoning / Science
arc-easy
ARC-Easy is the subset of grade-school science questions that a retrieval baseline or a word co-occurrence baseline answered correctly. It is the easier counterpart of ARC-Challenge and serves as the baseline measure of …
Paper: arXiv:1803.05457
Instruction following
arena-hard
Arena-Hard-Auto selects 500 prompts from 200,000 real LMArena user conversations. Answers are scored automatically with a strong model as judge, which is reported to correlate closely with human preference results.
Repository: Official repository
Agents / Search
assistantbench
AssistantBench collects realistic, time-consuming tasks that can only be answered by working through the live web, written so that each has one determinable answer. It measures whether a web agent is actually useful.
Paper: arXiv:2407.15711
Reasoning
bbeh
BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categorie…
Repository: Source repository
Reasoning
bbh
BIG-Bench Hard is the set of 23 BIG-Bench tasks on which models of the time scored below the average human rater. It is best known for showing how much chain-of-thought prompting changes the result.
Repository: Official repository
Agents / Tools
bfcl
The Berkeley Function Calling Leaderboard evaluates tool use. It works up from a single call to parallel calls, multi-turn dialogue, and recognising when no function should be called at all.
Code
bigcodebench
BigCodeBench poses 1,140 function-level tasks drawn across 139 libraries. It tests whether a model can compose several real APIs correctly rather than write one self-contained function.
Paper: arXiv:2406.15877
Reasoning
boolq
BoolQ pairs naturally occurring yes/no questions from search queries with a passage that decides them. The format is simple, but the inference it needs is consistently heavier than it looks.
Repository: Official repository
Agents / Search
browsecomp
BrowseComp evaluates web browsing and the ability to identify answers from multiple clues. This catalog supports introductions and discussion of search methods without republishing decrypted questions or answers.
Repository: Source repository
Agents / Search
browsecomp-plus
BrowseComp-Plus supports evaluation of retrieval methods using BrowseComp tasks and a search corpus. This catalog links to the official distribution without republishing plaintext questions or corpus content.
Paper: arXiv:2508.06600
Code
code-contests
CodeContests is the competitive-programming dataset built for AlphaCode, drawing problems from Codeforces and similar sites. Each problem ships with both correct and incorrect submissions and a large generated test suite…
Repository: Official repository
Science
critpt
CritPt evaluates scientific understanding and multi-step reasoning and computation on research-level physics problems. Its public dataset contains 70 challenges with problem descriptions and code templates.
Paper: arXiv:2509.26574
Code / Reasoning
cruxeval
CRUXEval takes 800 short Python functions and asks either for the output given an input (CRUXEval-O) or for an input that produces a given output (CRUXEval-I). It measures the ability to execute code mentally rather than…
Paper: arXiv:2401.03065
Reasoning
drop
DROP is a reading-comprehension benchmark whose questions require discrete operations over the paragraph — addition, subtraction, comparison, counting, sorting. It is built so that extracting a span cannot answer them.
Paper: arXiv:1903.00161
Code / Science
ds-1000
DS-1000 builds 1,000 data-science problems from real StackOverflow questions across seven libraries including NumPy, Pandas and Matplotlib. The problems are perturbed so that recalling the original answer does not solve …
Paper: arXiv:2211.11501
Mathematics
frontiermath
FrontierMath evaluates advanced mathematical reasoning using problems written by experts. This catalog links to the official introduction and public examples while excluding private evaluation problems.
Science
frontierscience
FrontierScience evaluates the ability to solve expert-level scientific tasks. Its public data separates olympiad and research problems so that competition and research tasks can be examined independently.
Agents / Multimodal
gaia
GAIA evaluates general AI assistants on tasks involving information gathering and tool use. Some tasks include attachments; this catalog provides an introduction in view of access and resharing restrictions.
Paper: arXiv:2311.12983
Science
gpqa
GPQA is a set of graduate-level questions in biology, physics and chemistry written so that non-experts with web access still fail them. The Diamond subset is the most-used slice.
Paper: arXiv:2311.12022
Mathematics
gsm8k
GSM8K is a set of grade-school word problems that each need several arithmetic steps. It remains a standard reference point for multi-step reasoning.
Paper: arXiv:2110.14168
Safety
harmbench
HarmBench is a standardized framework for comparing automated red-teaming attacks and refusal robustness on the same footing. It pits attack methods against defences and measures the success rate against harmful behaviou…
Paper: arXiv:2402.04249
Reasoning
hellaswag
HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.
Paper: arXiv:1905.07830
Code
hlce
Humanity's Last Code Exam evaluates problem solving and code generation using ICPC World Finals and IOI problems. The project describes 235 problems; this import selects its public ICPC dataset of 146 problems.
Repository: Source repository
Reasoning / Multimodal
hle
Humanity's Last Exam evaluates AI knowledge and reasoning across academic disciplines. It includes text and image questions; this catalog provides an introduction and official links in accordance with the publisher's req…
Code
humaneval
HumanEval asks for a Python function body from a docstring and signature, checked by hidden unit tests. It is the original functional-correctness benchmark for code generation.
Paper: arXiv:2107.03374
Code
humaneval-plus
HumanEval+ keeps the 164 original HumanEval problems and multiplies their tests by roughly 80×. It exists because the original suite was loose enough to pass implementations that were actually wrong.
Repository: Official repository
Reasoning / Instruction following
ifeval
IFEval measures whether a model follows verifiable instructions -- word counts, formats, forbidden words -- rather than whether an answer is good. Every constraint is checked programmatically.
Paper: arXiv:2311.07911
Safety
jailbreakbench
JailbreakBench is an open robustness benchmark for jailbreaking attacks, defining 100 harmful behaviours alongside 100 matched benign ones. This catalogue deliberately introduces and links it rather than reproducing any …
Paper: arXiv:2404.01318
Code
livecodebench
LiveCodeBench continuously collects new competitive-programming problems to evaluate coding capabilities. Its initial release_v1 contains 400 problems, with source platform, contest date, and dataset version tracked expl…
Paper: arXiv:2403.07974
Reasoning / Long context
longbench-v2
LongBench v2 evaluates deep understanding and reasoning over long contexts through multiple-choice questions. Its official description lists 503 questions spanning tasks such as single-document and multi-document QA and …
Paper: arXiv:2412.15204
Mathematics
math
MATH organises 12,500 high-school competition problems into seven subjects and five difficulty levels, each with a full worked solution. It underlies much of the later mathematics evaluation work; this catalogue imports …
Repository: Official repository
Mathematics / Multimodal
mathvista
MathVista gathers 6,141 mathematical problems that come with figures, charts or geometric diagrams, of which 1,000 form the public testmini split. It measures whether visual information can be carried into mathematical r…
Paper: arXiv:2310.02255
Code
mbpp
Mostly Basic Python Problems asks for a short Python function from a one-sentence description, checked by assertions. The sanitized split is the hand-verified subset.
Paper: arXiv:2108.07732
Code
mbpp-plus
MBPP+ takes the 378 problems of the sanitized MBPP set and multiplies their tests by roughly 35 times. It removes the same test-weakness inflation that HumanEval+ addresses.
Repository: Official repository
Mathematics / Multilingual
mgsm
MGSM is a human translation of 250 GSM8K problems into eleven languages, measuring whether reasoning survives the change of language. The gap against English is the multilingual signal.
Repository: Official repository
Reasoning / Knowledge
mmlu
MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.
Repository: Official repository
Reasoning
mmlu-pro
MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.
Paper: arXiv:2406.01574
Multimodal
mmmu
MMMU collects 11,500 college-level questions across six disciplines and thirty subjects, each carrying figures, diagrams or images that have to be read as well as reasoned over. MMMU-Pro is its tightened revision.
Paper: arXiv:2311.16502
Multimodal
mmmu-pro
MMMU-Pro evaluates multimodal understanding across academic disciplines using images and text. The official card lists 1,730 questions per setting, separating four-option and ten-option standard settings from the vision …
Paper: arXiv:2409.02813
Instruction following
mt-bench
MT-Bench is a set of 80 two-turn questions across eight categories, published with the study that established LLM-as-a-judge evaluation. It looks at multi-turn consistency and instruction following.
Repository: Official evaluation code
Reasoning
musr
MuSR poses multistep soft reasoning problems over algorithmically generated narratives. It has three domains — murder mysteries, object placements and team allocation — and this catalogue imports all three domains, 756 p…
Paper: arXiv:2310.16049
Mathematics / Science / Multimodal
olympiadbench
OlympiadBench evaluates scientific reasoning on Olympiad-level mathematics and physics problems. Its official description lists 8,476 English and Chinese problems with separate text-only and multimodal settings.
Repository: Official repository
Mathematics
omni-math
Omni-MATH evaluates mathematical reasoning on Olympiad-level problems. Its official dataset contains 4,428 problems accompanied by domain and difficulty information.
Paper: arXiv:2410.07985