benchmarks.wiki / Public workspace
Catalogue overview
Compare the public catalogue in one place.
Counts cover public problems with full text, not the size of the upstream datasets. An em dash means not recorded.
Highest recorded quotes the largest published score this catalogue has recorded for a benchmark. It is not a measurement by this site, and not a survey of everything published: a benchmark's own leaderboard may hold higher entries that are not recorded here. Evaluation conditions and metrics differ between reports, so this is not a ranking.
Answer availability describes what the source distributes; no answer values are shown. Awaiting a first attempt means no public discussion is recorded.
Try a random problem · Find a problem awaiting an attempt · Read this catalogue as JSON
| Benchmark | Field | Publication mode | Problems | Formats | Answer availability | Images | Highest recorded | Awaiting a first attempt |
|---|---|---|---|---|---|---|---|---|
| SciCode | Science · Code | Full text | 65 | scicode: 15 scicode-step: 50 | Source answer published: 65 | 0 | — | 65 |
| GPQA | Science | — | — | — | — | — | — | — |
| FrontierScience | Science | Full text | 100 | text: 100 | Source answer published: 100 | 0 | — | 100 |
| CritPt | Science | Full text | 70 | text: 70 | Source answer published: 70 | 0 | — | 70 |
| AgentHarm | Safety · Agents | — | — | — | — | — | — | — |
| JailbreakBench | Safety | — | — | — | — | — | — | — |
| HarmBench | Safety | — | — | — | — | — | — | — |
| ARC-Easy | Reasoning · Science | Full text | 2,376 | text: 2,376 | Source answer published: 2,376 | 0 | — | 2,376 |
| ARC-Challenge | Reasoning · Science | Full text | 1,172 | text: 1,172 | Source answer published: 1,172 | 0 | 95.3 Acc. · DeepSeek-V3Vendor self-report · Date not recorded · Base Model, Standard Benchmarks, ARC-Challenge (Acc.), final column | 1,172 |
| TruthfulQA | Reasoning · Safety | Full text | 817 | text: 817 | Source answer published: 817 | 0 | — | 817 |
| Humanity's Last Exam | Reasoning · Multimodal | — | — | — | — | — | 38.3 Accuracy (%) · Gemini 3 ProLeaderboard · Date not recorded · Quantitative Results, Accuracy column | — |
| RULER | Reasoning · Long context | — | — | — | — | — | — | — |
| LongBench v2 | Reasoning · Long context | Full text | 200 | text: 200 | Source answer published: 200 | 0 | — | 200 |
| MMLU | Reasoning · Knowledge | Full text | 14,015 | text: 14,015 | Source answer published: 14,015 | 0 | 87.1 Acc. · DeepSeek-V3Vendor self-report · Date not recorded · Base Model, Standard Benchmarks, MMLU (Acc.), final column | 14,015 |
| IFEval | Reasoning · Instruction following | Full text | 541 | text: 541 | — | 0 | — | 541 |
| ARC-AGI-3 | Reasoning · Agents | — | — | — | — | — | 0.37 Score (%) · Gemini 3.1 Pro PreviewPaper · Date not recorded · Table 2; Section 4.3.1 | — |
| ZebraLogicBench | Reasoning | Full text | 1,000 | text: 1,000 | — | 0 | — | 1,000 |
| WinoGrande | Reasoning | — | — | — | — | — | 84.9 Acc. · DeepSeek-V3Vendor self-report · Date not recorded · Base Model, Standard Benchmarks, WinoGrande (Acc.), final column | — |
| MuSR | Reasoning | Full text | 756 | text: 756 | Source answer published: 756 | 0 | — | 756 |
| MMLU-Pro | Reasoning | Full text | 12,032 | text: 12,032 | Source answer published: 12,032 | 0 | 64.4 Acc. · DeepSeek-V3Vendor self-report · Date not recorded · Base Model, Standard Benchmarks, MMLU-Pro (Acc.), final column | 12,032 |
| HellaSwag | Reasoning | Full text | 10,042 | text: 10,042 | Source answer published: 10,042 | 0 | 88.9 Acc. · DeepSeek-V3Vendor self-report · Date not recorded · Base Model, Standard Benchmarks, HellaSwag (Acc.), final column | 10,042 |
| DROP | Reasoning | Full text | 9,535 | text: 9,535 | Source answer published: 9,535 | 0 | 89 F1 · DeepSeek-V3Vendor self-report · Date not recorded · Base Model, Standard Benchmarks, DROP (F1), final column | 9,535 |
| BoolQ | Reasoning | Full text | 3,270 | text: 3,270 | Source answer published: 3,270 | 0 | — | 3,270 |
| BIG-Bench Hard | Reasoning | Full text | 6,507 | text: 6,507 | Source answer published: 6,507 | 0 | 87.5 EM · DeepSeek-V3Vendor self-report · Date not recorded · Base Model, Standard Benchmarks, BBH (EM), final column | 6,507 |
| BIG-Bench Extra Hard (BBEH) | Reasoning | Full text | 4,513 | text: 4,513 | Source answer published: 4,511 | 0 | — | 4,513 |
| ARC-AGI-2 | Reasoning | Full text | 120 | grid: 120 | Source answer published: 120 | 0 | — | 119 |
| MMMU-Pro | Multimodal | Full text | 200 | text: 200 | Source answer published: 200 | 240 | — | 200 |
| MMMU | Multimodal | — | — | — | — | — | — | — |
| OlympiadBench | Mathematics · Science · Multimodal | Full text | 200 | text: 200 | Source answer published: 200 | 0 | — | 200 |
| MathVista | Mathematics · Multimodal | — | — | — | — | — | — | — |
| MGSM | Mathematics · Multilingual | Full text | 2,750 | math: 2,750 | Source answer published: 2,750 | 0 | 79.8 EM · DeepSeek-V3Vendor self-report · Date not recorded · Base Model, Standard Benchmarks, MGSM (EM), final column | 2,750 |
| Omni-MATH | Mathematics | Full text | 4,406 | math: 4,206 text: 200 | Source answer published: 4,406 | 0 | — | 4,406 |
| MATH | Mathematics | Full text | 5,000 | math: 5,000 | Source answer published: 5,000 | 0 | 61.6 EM · DeepSeek-V3Vendor self-report · Date not recorded · Base Model, Standard Benchmarks, MATH (EM), final column | 5,000 |
| GSM8K | Mathematics | Full text | 1,319 | text: 1,319 | Source answer published: 1,319 | 0 | 89.3 EM · DeepSeek-V3Vendor self-report · Date not recorded · Base Model, Standard Benchmarks, GSM8K (EM), final column | 1,319 |
| FrontierMath | Mathematics | — | — | — | — | — | — | — |
| AMC (AIMO validation set) | Mathematics | Full text | 83 | math: 83 | Source answer published: 83 | 0 | — | 83 |
| AIME 2025 I | Mathematics | Full text | 15 | math: 15 | Source answer published: 15 | 0 | — | 15 |
| AIME 2024 | Mathematics | Full text | 30 | math: 30 | Source answer published: 30 | 0 | 39.2 Pass@1 · DeepSeek V3Vendor self-report · Date not recorded · Chat Model, Standard Benchmarks, AIME 2024 (Pass@1), final column | 30 |
| SimpleQA | Knowledge · Search | Full text | 4,326 | text: 4,326 | Source answer published: 4,325 | 0 | — | 4,326 |
| MT-Bench | Instruction following | — | — | — | — | — | — | — |
| Arena-Hard-Auto | Instruction following | — | — | — | — | — | — | — |
| DS-1000 | Code · Science | Full text | 1,000 | code: 1,000 | Source answer published: 1,000 | 0 | — | 1,000 |
| CRUXEval | Code · Reasoning | Full text | 800 | code: 800 | Source answer published: 800 | 0 | 69.8 Acc. · DeepSeek-V3Vendor self-report · Date not recorded · Base Model, Standard Benchmarks, CRUXEval-O (Acc.), final column | 800 |
| SWE-bench Verified | Code · Agents | — | — | — | — | — | 74.4 resolved (%) · Claude 4.5 Opus (20251101) (medium)Leaderboard · · Verified leaderboard; 20251124_mini-v1.16.0_claude-opus-4-5-20251101 | — |
| SWE-bench Lite | Code · Agents | — | — | — | — | — | 56.67 resolved (%) · SWE-agent + Claude 4 SonnetLeaderboard · · Lite leaderboard; 20250526_sweagent_claude-4-sonnet-20250514 | — |
| SWE-Bench Pro | Code · Agents | Full text | 200 | text: 200 | External grader: 200 | 0 | — | 200 |
| MBPP+ | Code | Full text | 378 | code: 378 | Source answer published: 378 | 0 | — | 378 |
| MBPP | Code | Full text | 1,401 | text: 1,401 | Source answer published: 1,401 | 0 | 75.4 Pass@1 · DeepSeek-V3Vendor self-report · Date not recorded · Base Model, Standard Benchmarks, MBPP (Pass@1), final column | 1,401 |
| LiveCodeBench | Code | Full text | 100 | text: 100 | External grader: 100 | 0 | — | 100 |
| Humanity's Last Code Exam | Code | Full text | 146 | text: 146 | Source answer published: 146 | 0 | — | 146 |
| HumanEval+ | Code | Full text | 164 | code: 164 | Source answer published: 164 | 0 | — | 164 |
| HumanEval | Code | Full text | 164 | text: 164 | Source answer published: 164 | 0 | 65.2 Pass@1 · DeepSeek-V3Vendor self-report · Date not recorded · Base Model, Standard Benchmarks, HumanEval (Pass@1), final column | 164 |
| CodeContests | Code | — | — | — | — | — | — | — |
| BigCodeBench | Code | Full text | 1,139 | code: 1,139 | Source answer published: 1,139 | 0 | — | 1,139 |
| Berkeley Function Calling Leaderboard | Agents · Tools | — | — | — | — | — | 77.47 Overall Acc (%) · Claude-Opus-4-5-20251101 (FC)Leaderboard · Date not recorded · Official BFCL V4 data, Overall Acc column | — |
| WebArena | Agents · Search | — | — | — | — | — | — | — |
| BrowseComp-Plus | Agents · Search | — | — | — | — | — | 55.9 Accuracy (%) · gpt-5Paper · Date not recorded · Table 1, BM25 rows | — |
| BrowseComp | Agents · Search | — | — | — | — | — | 9.9 Accuracy (%) · o1-2024-12-17Paper · Date not recorded · Table 3; Section 4.1 model identifiers | — |
| AssistantBench | Agents · Search | Full text | 181 | text: 181 | — | 0 | — | 181 |
| OSWorld | Agents · Multimodal | — | — | — | — | — | 12.24 Success Rate (%) · GPT-4Paper · Date not recorded · Table 5, Overall column; Section 4.1 | — |
| GAIA | Agents · Multimodal | — | — | — | — | — | 30.3 Score in % · GPT4 + plugins*Paper · Date not recorded · Table 4, Level 1 score | — |
| Terminal-Bench 2.1 | Agents · Code | Full text | 89 | text: 89 | External grader: 89 | 0 | — | 89 |
| tau-bench | Agents | — | — | — | — | — | — | — |