benchmarks.wiki / Public workspace
Catalogue overview
Compare the public catalogue in one place.
Counts cover public problems with full text, not the size of the upstream datasets. An em dash means not recorded.
Highest recorded quotes the largest published score this catalogue has recorded for a benchmark. It is not a measurement by this site, and not a survey of everything published: a benchmark's own leaderboard may hold higher entries that are not recorded here. Evaluation conditions and metrics differ between reports, so this is not a ranking.
Answer availability describes what the source distributes; no answer values are shown. Awaiting a first attempt means no public discussion is recorded.
Try a random problem · Find a problem awaiting an attempt · Read this catalogue as JSON
| Benchmark | Field | Publication mode | Problems | Formats | Answer availability | Images | Highest recorded | Awaiting a first attempt |
|---|---|---|---|---|---|---|---|---|
| tau-bench | Agents | — | — | — | — | — | — | — |
| Terminal-Bench 2.1 | Agents · Code | Full text | 89 | text: 89 | External grader: 89 | 0 | — | 89 |
| GAIA | Agents · Multimodal | — | — | — | — | — | 30.3 Score in % · GPT4 + plugins*Paper · Date not recorded · Table 4, Level 1 score | — |
| OSWorld | Agents · Multimodal | — | — | — | — | — | 12.24 Success Rate (%) · GPT-4Paper · Date not recorded · Table 5, Overall column; Section 4.1 | — |
| AssistantBench | Agents · Search | Full text | 181 | text: 181 | — | 0 | — | 181 |
| BrowseComp | Agents · Search | — | — | — | — | — | 9.9 Accuracy (%) · o1-2024-12-17Paper · Date not recorded · Table 3; Section 4.1 model identifiers | — |
| BrowseComp-Plus | Agents · Search | — | — | — | — | — | 55.9 Accuracy (%) · gpt-5Paper · Date not recorded · Table 1, BM25 rows | — |
| WebArena | Agents · Search | — | — | — | — | — | — | — |
| Berkeley Function Calling Leaderboard | Agents · Tools | — | — | — | — | — | 77.47 Overall Acc (%) · Claude-Opus-4-5-20251101 (FC)Leaderboard · Date not recorded · Official BFCL V4 data, Overall Acc column | — |
| BigCodeBench | Code | Full text | 1,139 | code: 1,139 | Source answer published: 1,139 | 0 | — | 1,139 |
| CodeContests | Code | — | — | — | — | — | — | — |
| HumanEval | Code | Full text | 164 | text: 164 | Source answer published: 164 | 0 | 65.2 Pass@1 · DeepSeek-V3Vendor self-report · Date not recorded · Base Model, Standard Benchmarks, HumanEval (Pass@1), final column | 164 |
| HumanEval+ | Code | Full text | 164 | code: 164 | Source answer published: 164 | 0 | — | 164 |
| Humanity's Last Code Exam | Code | Full text | 146 | text: 146 | Source answer published: 146 | 0 | — | 146 |
| LiveCodeBench | Code | Full text | 100 | text: 100 | External grader: 100 | 0 | — | 100 |
| MBPP | Code | Full text | 1,401 | text: 1,401 | Source answer published: 1,401 | 0 | 75.4 Pass@1 · DeepSeek-V3Vendor self-report · Date not recorded · Base Model, Standard Benchmarks, MBPP (Pass@1), final column | 1,401 |
| MBPP+ | Code | Full text | 378 | code: 378 | Source answer published: 378 | 0 | — | 378 |
| SWE-Bench Pro | Code · Agents | Full text | 200 | text: 200 | External grader: 200 | 0 | — | 200 |
| SWE-bench Lite | Code · Agents | — | — | — | — | — | 56.67 resolved (%) · SWE-agent + Claude 4 SonnetLeaderboard · · Lite leaderboard; 20250526_sweagent_claude-4-sonnet-20250514 | — |
| SWE-bench Verified | Code · Agents | — | — | — | — | — | 74.4 resolved (%) · Claude 4.5 Opus (20251101) (medium)Leaderboard · · Verified leaderboard; 20251124_mini-v1.16.0_claude-opus-4-5-20251101 | — |
| CRUXEval | Code · Reasoning | Full text | 800 | code: 800 | Source answer published: 800 | 0 | 69.8 Acc. · DeepSeek-V3Vendor self-report · Date not recorded · Base Model, Standard Benchmarks, CRUXEval-O (Acc.), final column | 800 |
| DS-1000 | Code · Science | Full text | 1,000 | code: 1,000 | Source answer published: 1,000 | 0 | — | 1,000 |
| Arena-Hard-Auto | Instruction following | — | — | — | — | — | — | — |
| MT-Bench | Instruction following | — | — | — | — | — | — | — |
| SimpleQA | Knowledge · Search | Full text | 4,326 | text: 4,326 | Source answer published: 4,325 | 0 | — | 4,326 |
| AIME 2024 | Mathematics | Full text | 30 | math: 30 | Source answer published: 30 | 0 | 39.2 Pass@1 · DeepSeek V3Vendor self-report · Date not recorded · Chat Model, Standard Benchmarks, AIME 2024 (Pass@1), final column | 30 |
| AIME 2025 I | Mathematics | Full text | 15 | math: 15 | Source answer published: 15 | 0 | — | 15 |
| AMC (AIMO validation set) | Mathematics | Full text | 83 | math: 83 | Source answer published: 83 | 0 | — | 83 |
| FrontierMath | Mathematics | — | — | — | — | — | — | — |
| GSM8K | Mathematics | Full text | 1,319 | text: 1,319 | Source answer published: 1,319 | 0 | 89.3 EM · DeepSeek-V3Vendor self-report · Date not recorded · Base Model, Standard Benchmarks, GSM8K (EM), final column | 1,319 |
| MATH | Mathematics | Full text | 5,000 | math: 5,000 | Source answer published: 5,000 | 0 | 61.6 EM · DeepSeek-V3Vendor self-report · Date not recorded · Base Model, Standard Benchmarks, MATH (EM), final column | 5,000 |
| Omni-MATH | Mathematics | Full text | 4,406 | math: 4,206 text: 200 | Source answer published: 4,406 | 0 | — | 4,406 |
| MGSM | Mathematics · Multilingual | Full text | 2,750 | math: 2,750 | Source answer published: 2,750 | 0 | 79.8 EM · DeepSeek-V3Vendor self-report · Date not recorded · Base Model, Standard Benchmarks, MGSM (EM), final column | 2,750 |
| MathVista | Mathematics · Multimodal | — | — | — | — | — | — | — |
| OlympiadBench | Mathematics · Science · Multimodal | Full text | 200 | text: 200 | Source answer published: 200 | 0 | — | 200 |
| MMMU | Multimodal | — | — | — | — | — | — | — |
| MMMU-Pro | Multimodal | Full text | 200 | text: 200 | Source answer published: 200 | 240 | — | 200 |
| ARC-AGI-2 | Reasoning | Full text | 120 | grid: 120 | Source answer published: 120 | 0 | — | 119 |
| BIG-Bench Extra Hard (BBEH) | Reasoning | Full text | 4,513 | text: 4,513 | Source answer published: 4,511 | 0 | — | 4,513 |
| BIG-Bench Hard | Reasoning | Full text | 6,507 | text: 6,507 | Source answer published: 6,507 | 0 | 87.5 EM · DeepSeek-V3Vendor self-report · Date not recorded · Base Model, Standard Benchmarks, BBH (EM), final column | 6,507 |
| BoolQ | Reasoning | Full text | 3,270 | text: 3,270 | Source answer published: 3,270 | 0 | — | 3,270 |
| DROP | Reasoning | Full text | 9,535 | text: 9,535 | Source answer published: 9,535 | 0 | 89 F1 · DeepSeek-V3Vendor self-report · Date not recorded · Base Model, Standard Benchmarks, DROP (F1), final column | 9,535 |
| HellaSwag | Reasoning | Full text | 10,042 | text: 10,042 | Source answer published: 10,042 | 0 | 88.9 Acc. · DeepSeek-V3Vendor self-report · Date not recorded · Base Model, Standard Benchmarks, HellaSwag (Acc.), final column | 10,042 |
| MMLU-Pro | Reasoning | Full text | 12,032 | text: 12,032 | Source answer published: 12,032 | 0 | 64.4 Acc. · DeepSeek-V3Vendor self-report · Date not recorded · Base Model, Standard Benchmarks, MMLU-Pro (Acc.), final column | 12,032 |
| MuSR | Reasoning | Full text | 756 | text: 756 | Source answer published: 756 | 0 | — | 756 |
| WinoGrande | Reasoning | — | — | — | — | — | 84.9 Acc. · DeepSeek-V3Vendor self-report · Date not recorded · Base Model, Standard Benchmarks, WinoGrande (Acc.), final column | — |
| ZebraLogicBench | Reasoning | Full text | 1,000 | text: 1,000 | — | 0 | — | 1,000 |
| ARC-AGI-3 | Reasoning · Agents | — | — | — | — | — | 0.37 Score (%) · Gemini 3.1 Pro PreviewPaper · Date not recorded · Table 2; Section 4.3.1 | — |
| IFEval | Reasoning · Instruction following | Full text | 541 | text: 541 | — | 0 | — | 541 |
| MMLU | Reasoning · Knowledge | Full text | 14,015 | text: 14,015 | Source answer published: 14,015 | 0 | 87.1 Acc. · DeepSeek-V3Vendor self-report · Date not recorded · Base Model, Standard Benchmarks, MMLU (Acc.), final column | 14,015 |
| LongBench v2 | Reasoning · Long context | Full text | 200 | text: 200 | Source answer published: 200 | 0 | — | 200 |
| RULER | Reasoning · Long context | — | — | — | — | — | — | — |
| Humanity's Last Exam | Reasoning · Multimodal | — | — | — | — | — | 38.3 Accuracy (%) · Gemini 3 ProLeaderboard · Date not recorded · Quantitative Results, Accuracy column | — |
| TruthfulQA | Reasoning · Safety | Full text | 817 | text: 817 | Source answer published: 817 | 0 | — | 817 |
| ARC-Challenge | Reasoning · Science | Full text | 1,172 | text: 1,172 | Source answer published: 1,172 | 0 | 95.3 Acc. · DeepSeek-V3Vendor self-report · Date not recorded · Base Model, Standard Benchmarks, ARC-Challenge (Acc.), final column | 1,172 |
| ARC-Easy | Reasoning · Science | Full text | 2,376 | text: 2,376 | Source answer published: 2,376 | 0 | — | 2,376 |
| HarmBench | Safety | — | — | — | — | — | — | — |
| JailbreakBench | Safety | — | — | — | — | — | — | — |
| AgentHarm | Safety · Agents | — | — | — | — | — | — | — |
| CritPt | Science | Full text | 70 | text: 70 | Source answer published: 70 | 0 | — | 70 |
| FrontierScience | Science | Full text | 100 | text: 100 | Source answer published: 100 | 0 | — | 100 |
| GPQA | Science | — | — | — | — | — | — | — |
| SciCode | Science · Code | Full text | 65 | scicode: 15 scicode-step: 50 | Source answer published: 65 | 0 | — | 65 |