{"items":[{"benchmark":{"slug":"tau-bench","formal_name":"tau-bench","category":"Agents","effective_mode":"metadata_only","profile":{"resources":[{"kind":"repository","url":"https://github.com/sierra-research/tau-bench","label":"Official repository"},{"kind":"issues","url":"https://github.com/sierra-research/tau-bench/issues","label":"Issue tracker"}],"task_format":"A conversational task with a user simulator, a set of tools and a domain policy the agent must respect.","scoring":"Tasks are executed against a simulated user and a live tool API rather than answered from text, so there is no problem statement to republish.","metric":"","size":"Not imported here; see the official distribution for the current size.","answer_access":"Tasks are executed against a simulated user and a live tool API rather than answered from text, so there is no problem statement to republish.","license":"MIT (declared on the source)","history":"Sierra researchers introduced tau-bench in 2024 to evaluate agents that interact with users and tools while following domain policies. The motivating problem was reliable completion of customer-service tasks across multiple conversational turns. The official repository recorded in this catalog was created in June 2024.","construction":"Each task combines a simulated user, a domain policy and an API through which the agent can inspect or change the environment. The agent works through a conversation and tool calls, and evaluation compares the resulting database state with the task's expected outcome. This produces an executable interaction task rather than a static question with a reference sentence.","limitations":"The user simulator and tool environment are part of the evaluation, so changes to either can change the observed success rate. Matching the expected database state establishes an outcome but does not by itself establish that every conversational statement was correct or every intermediate action appropriate. Customer-service policies and APIs define a bounded domain, limiting the conclusions that can be drawn about general autonomous work.","saturation":"","citation":"@misc{yao2024tau,\n      title={$\\tau$-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains}, \n      author={Shunyu Yao and Noah Shinn and Pedram Razavi and Karthik Narasimhan},\n      year={2024},\n      eprint={2406.12045},\n      archivePrefix={arXiv},\n      primaryClass={cs.AI},\n      url={https://arxiv.org/abs/2406.12045}, \n}\n@misc{barres2025tau2,\n      title={$\\tau^2$-Bench: Evaluating Conversational Agents in a Dual-Control Environment}, \n      author={Victor Barres and Honghua Dong and Soham Ray and Xujie Si and Karthik Narasimhan},\n      year={2025},\n      eprint={2506.07982},\n      archivePrefix={arXiv},\n      primaryClass={cs.AI},\n      url={https://arxiv.org/abs/2506.07982}, \n}","maintainer":"sierra-research (source organisation)","released":"Repository created on GitHub 2024-06","why_hard":"Success depends on multi-turn dialogue, tool use and rule compliance together; the score is the final database state, not the transcript.","related":["arc-agi-3","browsecomp","browsecomp-plus"],"scores":[]}},"facts":null},{"benchmark":{"slug":"terminal-bench-2-1","formal_name":"Terminal-Bench 2.1","category":"Agents / Code","effective_mode":"full","profile":{"resources":[{"kind":"repository","url":"https://github.com/harbor-framework/terminal-bench-2-1","label":"Source repository"},{"kind":"issues","url":"https://github.com/harbor-framework/terminal-bench-2-1/issues","label":"Repository issues"}],"task_format":"Each instruction.md becomes a task with an instruction field. The task also has environment configuration; the instruction import does not turn environment files into question text.","scoring":"Verification requires the external task environment and grader.","metric":"","size":"Catalog selection for version 2.1: tasks split, expected 89 task records, source-row limit 200.","answer_access":"No inline answer is acquired from instruction.md; verification requires the external task environment and grader.","license":"Apache-2.0 (declared in the GitHub repository)","history":"The harbor-framework organisation hosts Terminal-Bench 2.1 in a repository created in May 2026. This version evaluates agents carrying out tasks in terminal environments, where success requires interaction with a configured environment. It is tracked separately from Terminal-Bench 2.0, preserving the distinction between benchmark versions.","construction":"Each task supplies an instruction file and environment configuration, with verification performed through the external task environment and grader. The catalogue selection expects 89 task records for version 2.1; this is an acquisition expectation rather than a confirmed count of imported tasks. Importing the instructions alone does not reproduce the executable evaluation.","limitations":"A reported result depends on the agent, its tools and resource limits, and the configured environment as well as the underlying language model. The grader establishes success under the task checks, which need not capture every property of a robust real-world solution. Comparisons require the benchmark version and execution setup to be identified; a score for version 2.0 should not be substituted for a version 2.1 result.","saturation":"","citation":"@misc{merrill2026terminalbenchbenchmarkingagentshard,\n      title={Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces},\n      author={Mike A. Merrill and Alexander G. Shaw and Nicholas Carlini and Boxuan Li and Harsh Raj and Ivan Bercovich and Lin Shi and Jeong Yeon Shin and Thomas Walshe and E. Kelly Buchanan and Junhong Shen and Guanghao Ye and Haowei Lin and Jason Poulos and Maoyu Wang and Marianna Nezhurina and Jenia Jitsev and Di Lu and Orfeas Menis Mastromichalakis and Zhiwei Xu and Zizhao Chen and Yue Liu and Robert Zhang and Leon Liangyu Chen and Anurag Kashyap and Jan-Lucas Uslu and Jeffrey Li and Jianbo Wu and Minghao Yan and Song Bian and Vedang Sharma and Ke Sun and Steven Dillmann and Akshay Anand and Andrew Lanpouthakoun and Bardia Koopah and Changran Hu and Etash Guha and Gabriel H. S. Dreiman and Jiacheng Zhu and Karl Krauth and Li Zhong and Niklas Muennighoff and Robert Amanfu and Shangyin Tan and Shreyas Pimpalgaonkar and Tushar Aggarwal and Xiangning Lin and Xin Lan and Xuandong Zhao and Yiqing Liang and Yuanli Wang and Zilong Wang and Changzhi Zhou and David Heineman and Hange Liu and Harsh Trivedi and John Yang and Junhong Lin and Manish Shetty and Michael Yang and Nabil Omi and Negin Raoof and Shanda Li and Terry Yue Zhuo and Wuwei Lin and Yiwei Dai and Yuxin Wang and Wenhao Chai and Shang Zhou and Dariush Wahdany and Ziyu She and Jiaming Hu and Zhikang Dong and Yuxuan Zhu and Sasha Cui and Ahson Saiyed and Arinbjörn Kolbeinsson and Jesse Hu and Christopher Michael Rytting and Ryan Marten and Yixin Wang and Alex Dimakis and Andy Konwinski and Ludwig Schmidt},\n      year={2026},\n      eprint={2601.11868},\n      archivePrefix={arXiv},\n      primaryClass={cs.SE},\n      url={https://arxiv.org/abs/2601.11868},\n}","maintainer":"harbor-framework (GitHub organisation)","released":"Repository created on GitHub 2026-05","why_hard":"Requires performing tasks in terminal environments.","related":["swe-bench-pro","hlce","scicode"],"scores":[]}},"facts":{"slug":"terminal-bench-2-1","task_count":89,"image_count":0,"formats":[{"count":89,"value":"text"}],"answers":[{"count":89,"value":"external_grader"}],"configs":[{"count":89,"value":""}],"splits":[{"count":89,"value":"tasks"}],"waiting_count":89,"post_count":0,"discussed_count":0,"releases":[{"acquired":"2026-09-09","revision":"7131e4375048a0e408a8fb404b5f499d726b695b","release_id":"9bc97733b3bd586afbf8b4c003048d2f4216eeea2fce12e1800c9c6b2c3f8ff4"},{"acquired":"2026-09-09","revision":"7131e4375048a0e408a8fb404b5f499d726b695b","release_id":"f19a7a38c725f71084bb1fba61460af314acc298022aff01f26e10658b36ba43"},{"acquired":"2026-09-09","revision":"7131e4375048a0e408a8fb404b5f499d726b695b","release_id":"e3284554d77a84c2f305276ed068a412e6aab0786043491c98324b77e7ea2c61"},{"acquired":"2026-09-09","revision":"7131e4375048a0e408a8fb404b5f499d726b695b","release_id":"1a25552e232777089fb9cd9f11c6b4ea5b073ac1cc019329cb3ac8a4487e2433"}],"terms":{"license":null,"checked_at":null,"distribution_terms":null,"publisher_requests":null}}},{"benchmark":{"slug":"gaia","formal_name":"GAIA","category":"Agents / Multimodal","effective_mode":"metadata_only","profile":{"resources":[{"kind":"dataset","url":"https://huggingface.co/datasets/gaia-benchmark/GAIA","label":"Source dataset"},{"kind":"issues","url":"https://huggingface.co/datasets/gaia-benchmark/GAIA/discussions","label":"Dataset discussions"},{"kind":"paper","url":"https://arxiv.org/abs/2311.12983","label":"arXiv:2311.12983"}],"task_format":"Information-gathering and tool-use tasks; some include attachments.","scoring":"","metric":"","size":"","answer_access":"This catalog has not attempted authenticated acquisition; its acquisition record notes gated access and resharing restrictions.","license":"","history":"The GAIA research group introduced the benchmark in 2023 to evaluate general AI assistants on tasks requiring information gathering and tool use. The dataset was published under the gaia-benchmark organization on Hugging Face in October 2023. Its tasks connect language understanding to retrieving information and handling supporting materials, including attachments.","construction":"","limitations":"GAIA scores reflect the complete assistant setup, including access to tools and the ability to process attachments, rather than the language model alone. The paper's recorded GPT-4 result used manually selected plugins, and the source explicitly states that those results cannot be reproduced exactly. A result on Level 1 also cannot be treated as an overall score across the benchmark's difficulty levels.","saturation":"The recorded paper results on the 146 Level 1 questions report 30.3% for GPT-4 with manually selected plugins and 14.4% for AutoGPT with a GPT-4 backend. Those experiments left considerable headroom even at Level 1. They are historical results under specific tool configurations and do not establish how close later systems are to saturating GAIA.","citation":"","maintainer":"gaia-benchmark (Hugging Face organisation)","released":"Dataset published on Hugging Face 2023-10","why_hard":"Requires gathering information and using tools, including tasks with attachments.","related":["hle","swe-bench-pro","terminal-bench-2-1"],"scores":[{"model":"GPT4 + plugins*","organization":"","score":30.3,"metric":"Score in %","setup":"Level 1 (146 questions). Plugins manually selected; source says results cannot be reproduced exactly.","reported_by":"paper","source_url":"https://arxiv.org/html/2311.12983v1","source_label":"Table 4, Level 1 score","reported_at":"","checked_at":"2026-09-16"},{"model":"AutoGPT (GPT4 backend)","organization":"","score":14.4,"metric":"Score in %","setup":"Level 1 (146 questions).","reported_by":"paper","source_url":"https://arxiv.org/html/2311.12983v1","source_label":"Table 4, Level 1 score","reported_at":"","checked_at":"2026-09-16"}]}},"facts":null},{"benchmark":{"slug":"osworld","formal_name":"OSWorld","category":"Agents / Multimodal","effective_mode":"metadata_only","profile":{"resources":[{"kind":"repository","url":"https://github.com/xlang-ai/OSWorld","label":"Official repository"},{"kind":"issues","url":"https://github.com/xlang-ai/OSWorld/issues","label":"Issue tracker"}],"task_format":"A natural-language task in a real desktop environment, with a setup script and an execution-based checker.","scoring":"Every task is executed in a virtual machine and graded by an execution script; there is no text corpus to republish.","metric":"","size":"Not imported here; see the official distribution for the current size.","answer_access":"Every task is executed in a virtual machine and graded by an execution script; there is no text corpus to republish.","license":"Apache-2.0 (declared on the source)","history":"The xlang-ai research group introduced OSWorld in 2024 to evaluate agents performing tasks in real desktop operating systems. It addressed the gap between answering questions about an interface and actually carrying out work across applications and files. The October 2023 date recorded for this source is the GitHub repository creation date, rather than the date of the benchmark paper.","construction":"OSWorld tasks pair a natural-language request with an initial desktop state established by a setup script. An agent operates inside a virtual machine, and an execution-based checker examines whether the requested outcome was achieved. This makes the benchmark a collection of executable environments and task-specific checks, rather than a static set of questions with textual reference answers.","limitations":"Success depends on the agent harness as well as the underlying model, including the observation format, action interface and allowed number of steps. Application versions and environment state can also affect whether an interaction succeeds. Execution-based checking captures the conditions encoded by each checker, rather than every aspect of the quality of the finished work. Results using accessibility trees and results using screenshots therefore describe materially different settings.","saturation":"The paper results recorded here report 12.24 percent success for GPT-4 with accessibility-tree observations and 5.26 percent for GPT-4V with screenshots, both under a 15-step limit on Ubuntu tasks. These historical results leave substantial headroom in those configurations. They do not establish current best performance, and the different observation formats prevent a clean model-only comparison.","citation":"@misc{OSWorld,\n      title={OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments}, \n      author={Tianbao Xie and Danyang Zhang and Jixuan Chen and Xiaochuan Li and Siheng Zhao and Ruisheng Cao and Toh Jing Hua and Zhoujun Cheng and Dongchan Shin and Fangyu Lei and Yitao Liu and Yiheng Xu and Shuyan Zhou and Silvio Savarese and Caiming Xiong and Victor Zhong and Tao Yu},\n      year={2024},\n      eprint={2404.07972},\n      archivePrefix={arXiv},\n      primaryClass={cs.AI}\n}","maintainer":"xlang-ai (source organisation)","released":"Repository created on GitHub 2023-10","why_hard":"The agent has to operate real applications through the screen, and partial progress scores nothing.","related":["gaia","arc-agi-3","browsecomp"],"scores":[{"model":"GPT-4","organization":"","score":12.24,"metric":"Success Rate (%)","setup":"A11y tree; overall Ubuntu tasks; max 15 steps; temperature 1.0; top-p 0.9.","reported_by":"paper","source_url":"https://arxiv.org/html/2404.07972v1","source_label":"Table 5, Overall column; Section 4.1","reported_at":"","checked_at":"2026-09-16"},{"model":"GPT-4V","organization":"","score":5.26,"metric":"Success Rate (%)","setup":"Screenshot; overall Ubuntu tasks; max 15 steps; temperature 1.0; top-p 0.9.","reported_by":"paper","source_url":"https://arxiv.org/html/2404.07972v1","source_label":"Table 5, Overall column; Section 4.1","reported_at":"","checked_at":"2026-09-16"},{"model":"Claude-3-Opus","organization":"","score":2.42,"metric":"Success Rate (%)","setup":"Screenshot; overall Ubuntu tasks; max 15 steps; temperature 1.0; top-p 0.9.","reported_by":"paper","source_url":"https://arxiv.org/html/2404.07972v1","source_label":"Table 5, Overall column; Section 4.1","reported_at":"","checked_at":"2026-09-16"}]}},"facts":null},{"benchmark":{"slug":"assistantbench","formal_name":"AssistantBench","category":"Agents / Search","effective_mode":"full","profile":{"resources":[{"kind":"dataset","url":"https://huggingface.co/datasets/AssistantBench/AssistantBench","label":"Source dataset"},{"kind":"issues","url":"https://huggingface.co/datasets/AssistantBench/AssistantBench/discussions","label":"Dataset discussions"},{"kind":"site","url":"https://assistantbench.github.io/","label":"Official site"},{"kind":"paper","url":"https://arxiv.org/abs/2407.15711","label":"arXiv:2407.15711"}],"task_format":"One task statement. The evaluation subset, the answer, the gold source URL and the explanation are not part of the prompt.","scoring":"The test split ships no answer at all — the source withholds it — so this catalogue has nothing to store or render for it.","metric":"accuracy and answer rate over the held-out test set, scored by the official submission service","size":"All 181 tasks imported from the default / test selection, with a source-row limit of 200.","answer_access":"Columns outside the prompt allowlist, including any answer or reference solution, are separated into restricted storage and never rendered.","license":"apache-2.0 (declared on the Hugging Face dataset card)","history":"The AssistantBench research team released its dataset in July 2024 to evaluate assistants on realistic, time-consuming web tasks. Its motivating question was whether an agent could assemble a determinate answer through substantial search across multiple sources, rather than merely retrieve a fact from one page.","construction":"The benchmark collects tasks designed to have a determinable answer even when obtaining it requires work across websites. The source test selection contains 181 tasks and withholds their answers; the catalog imports all 181 task statements. Official evaluation uses a submission service and reports accuracy and answer rate on the held-out test set.","limitations":"Performance depends on the agent's search and browsing tools as well as the underlying model. Because tasks use the live web, changes in page availability and content can change the evidence accessible during an evaluation. Accuracy and answer rate describe different aspects of performance: a result needs to establish both how often an agent answers and how often it is correct.","saturation":"","citation":"@article{yoran2024assistantbench,\n  title={AssistantBench: Can Web Agents Solve Realistic and Time-Consuming Tasks?},\n  author={Ori Yoran and Samuel Joseph Amouyal and Chaitanya Malaviya and Ben Bogin and Ofir Press and Jonathan Berant},\n  journal={arXiv preprint arXiv:2407.15711},\n  year={2024}\n}","maintainer":"AssistantBench (source organisation)","released":"Dataset published on Hugging Face 2024-07","why_hard":"The answer is not on any single page. It has to be assembled across sites, and the search itself is most of the work.","related":["browsecomp","gaia","webarena"],"scores":[]}},"facts":{"slug":"assistantbench","task_count":181,"image_count":0,"formats":[{"count":181,"value":"text"}],"answers":[{"count":181,"value":"unavailable"}],"configs":[{"count":181,"value":"default"}],"splits":[{"count":181,"value":"test"}],"waiting_count":181,"post_count":0,"discussed_count":0,"releases":[{"acquired":"2026-09-12","revision":"482cbbc0400f6d048438c4021727f21a10cbff49","release_id":"02c88c74db5ac471a0e72e5fd3b9203c883c144ef47f800f72e7292ebd69a390"},{"acquired":"2026-09-12","revision":"482cbbc0400f6d048438c4021727f21a10cbff49","release_id":"868936e7972d66ff1eef7ba6ca23706cdc92790827bb8fd07a5180f216751260"},{"acquired":"2026-09-12","revision":"482cbbc0400f6d048438c4021727f21a10cbff49","release_id":"75ac7aefce1da9773f57e0bd77efc57a9747ca2b666ad8e3ee189cbe596d1ece"}],"terms":{"license":null,"checked_at":null,"distribution_terms":null,"publisher_requests":null}}},{"benchmark":{"slug":"browsecomp","formal_name":"BrowseComp","category":"Agents / Search","effective_mode":"metadata_only","profile":{"resources":[{"kind":"repository","url":"https://github.com/openai/simple-evals","label":"Source repository"},{"kind":"issues","url":"https://github.com/openai/simple-evals/issues","label":"Repository issues"},{"kind":"site","url":"https://openai.com/index/browsecomp/","label":"Official site"}],"task_format":"Web-browsing questions that require identifying an answer from multiple clues.","scoring":"","metric":"","size":"","answer_access":"This catalog has not attempted decryption and does not republish decrypted questions or answers.","license":"MIT (declared in the GitHub repository)","history":"OpenAI introduced BrowseComp in 2025 to test agents on difficult web searches whose answers must be identified from multiple clues. The benchmark targets the persistence and information gathering needed to locate obscure facts, rather than only the ability to answer familiar factual questions from memory.","construction":"","limitations":"A BrowseComp result reflects the browsing system, available tools and search budget in addition to the model. Results obtained without browsing therefore measure a different setup from results obtained with web access. Its focus on identifying answers from clues does not directly measure broader browser tasks such as completing transactions or editing documents.","saturation":"The paper results recorded here report 9.9% accuracy for o1 without browsing at medium reasoning effort, 1.9% for GPT-4o search preview with browsing, and 0.6% for GPT-4o without browsing. These particular results are far from the ceiling and show considerable remaining difficulty for the evaluated systems. They are not a current leaderboard maximum, and their different browsing setups prevent treating them as a controlled comparison of model capability alone.","citation":"","maintainer":"openai (GitHub organisation)","released":"Repository created on GitHub 2024-04","why_hard":"Requires finding information on the web and combining multiple clues.","related":["browsecomp-plus","swe-bench-pro","terminal-bench-2-1"],"scores":[{"model":"o1-2024-12-17","organization":"OpenAI","score":9.9,"metric":"Accuracy (%)","setup":"Without browsing; medium reasoning effort.","reported_by":"paper","source_url":"https://arxiv.org/html/2504.12516v1","source_label":"Table 3; Section 4.1 model identifiers","reported_at":"","checked_at":"2026-09-16"},{"model":"gpt-4o-search-preview-2025-03-11","organization":"OpenAI","score":1.9,"metric":"Accuracy (%)","setup":"With browsing.","reported_by":"paper","source_url":"https://arxiv.org/html/2504.12516v1","source_label":"Table 3; Section 4.1 model identifiers","reported_at":"","checked_at":"2026-09-16"},{"model":"gpt-4o-2024-08-06","organization":"OpenAI","score":0.6,"metric":"Accuracy (%)","setup":"Without browsing.","reported_by":"paper","source_url":"https://arxiv.org/html/2504.12516v1","source_label":"Table 3; Section 4.1 model identifiers","reported_at":"","checked_at":"2026-09-16"}]}},"facts":null},{"benchmark":{"slug":"browsecomp-plus","formal_name":"BrowseComp-Plus","category":"Agents / Search","effective_mode":"metadata_only","profile":{"resources":[{"kind":"dataset","url":"https://huggingface.co/datasets/Tevatron/browsecomp-plus","label":"Source dataset"},{"kind":"issues","url":"https://huggingface.co/datasets/Tevatron/browsecomp-plus/discussions","label":"Dataset discussions"},{"kind":"paper","url":"https://arxiv.org/abs/2508.06600","label":"arXiv:2508.06600"}],"task_format":"BrowseComp tasks paired with a search corpus for retrieval evaluation.","scoring":"","metric":"","size":"","answer_access":"This catalog does not republish plaintext questions or corpus content. Its acquisition record reports HTTP 401 for the old source; the official dataset link points to Tevatron/browsecomp-plus.","license":"mit (declared on the Hugging Face dataset card)","history":"BrowseComp-Plus adapted BrowseComp tasks for retrieval evaluation against a search corpus. The Tevatron organisation published the dataset on Hugging Face in August 2025. Pairing tasks with a fixed corpus makes it possible to study search and answering within a specified collection of documents.","construction":"","limitations":"A result on a fixed search corpus describes performance within that collection, rather than unrestricted browsing of the changing web. End-to-end answer accuracy combines retrieval, tool use and answer generation, so it does not isolate the quality of the language model or retriever alone. Comparisons require the same corpus and a clearly specified retrieval setup.","saturation":"The paper results recorded here report 55.9% accuracy for GPT-5, 49.28% for o3 and 14.58% for GPT-4.1, using a BM25 retriever, the same tool-use prompt and the fixed BrowseComp-Plus corpus. These results leave substantial headroom in that configuration. They do not establish a ceiling across other retrieval methods or agent configurations.","citation":"@article{chen2025BrowseCompPlus,\n      title={BrowseComp-Plus: A More Fair and Transparent Evaluation Benchmark of Deep-Research Agent}, \n      author={Chen, Zijian and Ma, Xueguang and Zhuang, Shengyao and Nie, Ping and Zou, Kai and Liu, Andrew and Green, Joshua and Patel, Kshama and Meng, Ruoxi and Su, Mingyi and Sharifymoghaddam, Sahel and Li, Yanxi and Hong, Haoran and Shi, Xinyu and Liu, Xuye and Thakur, Nandan and Zhang, Crystina and Gao, Luyu and Chen, Wenhu and Lin, Jimmy},\n      year={2025},\n      journal={arXiv preprint arXiv:2508.06600}\n}","maintainer":"Tevatron (Hugging Face organisation)","released":"Dataset published on Hugging Face 2025-08","why_hard":"Requires retrieving information for BrowseComp tasks from a search corpus.","related":["browsecomp","swe-bench-pro","terminal-bench-2-1"],"scores":[{"model":"gpt-5","organization":"","score":55.9,"metric":"Accuracy (%)","setup":"BM25 retriever; same tool-use prompt; fixed BrowseComp-Plus corpus.","reported_by":"paper","source_url":"https://arxiv.org/html/2508.06600v1","source_label":"Table 1, BM25 rows","reported_at":"","checked_at":"2026-09-16"},{"model":"o3","organization":"","score":49.28,"metric":"Accuracy (%)","setup":"BM25 retriever; same tool-use prompt; fixed BrowseComp-Plus corpus.","reported_by":"paper","source_url":"https://arxiv.org/html/2508.06600v1","source_label":"Table 1, BM25 rows","reported_at":"","checked_at":"2026-09-16"},{"model":"gpt-4.1","organization":"","score":14.58,"metric":"Accuracy (%)","setup":"BM25 retriever; same tool-use prompt; fixed BrowseComp-Plus corpus.","reported_by":"paper","source_url":"https://arxiv.org/html/2508.06600v1","source_label":"Table 1, BM25 rows","reported_at":"","checked_at":"2026-09-16"}]}},"facts":null},{"benchmark":{"slug":"webarena","formal_name":"WebArena","category":"Agents / Search","effective_mode":"metadata_only","profile":{"resources":[{"kind":"repository","url":"https://github.com/web-arena-x/webarena","label":"Official repository"},{"kind":"issues","url":"https://github.com/web-arena-x/webarena/issues","label":"Issue tracker"}],"task_format":"A natural-language goal on a self-hosted web application, with a programmatic check of the resulting state.","scoring":"Tasks run against self-hosted web applications and are graded by the resulting state, so there is no static problem text to republish.","metric":"","size":"Not imported here; see the official distribution for the current size.","answer_access":"Tasks run against self-hosted web applications and are graded by the resulting state, so there is no static problem text to republish.","license":"Apache-2.0 (declared on the source)","history":"The WebArena research team introduced the benchmark in 2023 to evaluate agents carrying out natural-language goals through web interfaces. The official web-arena-x repository was created in July 2023. Self-hosted applications made it possible to evaluate multi-step navigation and interaction in a controlled web environment.","construction":"WebArena uses self-hosted web applications representing settings such as shopping, discussion forums and code hosting, with supporting information resources. Human-authored tasks specify goals that agents must accomplish by interacting with those applications. Programmatic evaluators check task outcomes, including the resulting application state, rather than requiring the agent to follow a single reference sequence of clicks.","limitations":"The self-hosted environment provides controlled tasks but does not reproduce the full variability of the live web, including changing layouts, authentication barriers and service outages. A success rate also depends on the browser interface, available observations, action limits and the agent framework. Programmatic outcome checks cover specified conditions rather than every aspect of usability or safe interaction. Environment versions and reset procedures therefore matter when comparing published runs.","saturation":"","citation":"@article{zhou2023webarena,\n  title={WebArena: A Realistic Web Environment for Building Autonomous Agents},\n  author={Zhou, Shuyan and Xu, Frank F and Zhu, Hao and Zhou, Xuhui and Lo, Robert and Sridhar, Abishek and Cheng, Xianyi and Bisk, Yonatan and Fried, Daniel and Alon, Uri and others},\n  journal={arXiv preprint arXiv:2307.13854},\n  year={2023}\n}","maintainer":"web-arena-x (source organisation)","released":"Repository created on GitHub 2023-07","why_hard":"Long horizons on real interfaces: navigation, forms and search all have to work before the goal state can be reached.","related":["browsecomp","browsecomp-plus","arc-agi-3"],"scores":[]}},"facts":null},{"benchmark":{"slug":"bfcl","formal_name":"Berkeley Function Calling Leaderboard","category":"Agents / Tools","effective_mode":"metadata_only","profile":{"resources":[{"kind":"dataset","url":"https://huggingface.co/datasets/gorilla-llm/Berkeley-Function-Calling-Leaderboard","label":"Source dataset"},{"kind":"issues","url":"https://huggingface.co/datasets/gorilla-llm/Berkeley-Function-Calling-Leaderboard/discussions","label":"Dataset discussions"},{"kind":"leaderboard","url":"https://gorilla.cs.berkeley.edu/leaderboard.html","label":"Official leaderboard"}],"task_format":"A user request with a set of callable function signatures; the expected output is the call, not prose.","scoring":"Scored by abstract-syntax-tree comparison against the expected call, and by executing the call where the category allows it.","metric":"","size":"Several thousand evaluation cases across the categories; none imported.","answer_access":"No problem text is imported, so there is no answer storage for this entry. The source distributes the answers under its own terms.","license":"apache-2.0 (declared on the Hugging Face dataset card)","history":"The Gorilla team at UC Berkeley developed the Berkeley Function Calling Leaderboard to evaluate whether language models could select and invoke tools correctly. Its Hugging Face dataset was published in February 2024. The benchmark extends the evaluation of generated text to structured calls, including argument selection and cases in which no function should be called.","construction":"The source distribution contains several thousand evaluation cases organized into tool-use categories. Cases pair user requests with callable function signatures and expected calls, with settings covering single calls, parallel calls, multi-turn interaction and abstention from calling a function. Evaluation compares abstract syntax trees against expected calls and executes calls where the category supports execution.","limitations":"A correct function call demonstrates success under the supplied tool schema, rather than reliable operation in every external service that might expose a similar tool. Abstract-syntax-tree matching and execution-based checks test different aspects of correctness, so an overall score can conceal category-specific weaknesses. Comparisons also depend on the benchmark version and whether a model uses native function calling or a prompted representation of calls.","saturation":"The recorded BFCL V4 leaderboard reports overall accuracy of 77.47% for Claude Opus 4.5 with native function calling, followed by 73.24% for Claude Sonnet 4.5 in that mode. Gemini 3 Pro Preview is recorded at 72.51% using prompting. These results leave substantial overall headroom, but the different calling modes and the mixture of categories limit what a single ranking says about a particular tool-use workflow.","citation":"","maintainer":"Gorilla LLM, UC Berkeley (source organisation)","released":"Dataset published on Hugging Face 2024-02","why_hard":"Getting the function right is not enough — the arguments, their types and the decision not to call anything all count, and the last of those is where most models lose points.","related":["tau-bench","osworld","terminal-bench-2-1"],"scores":[{"model":"Claude-Opus-4-5-20251101 (FC)","organization":"Anthropic","score":77.47,"metric":"Overall Acc (%)","setup":"BFCL V4; overall; FC.","reported_by":"leaderboard","source_url":"https://gorilla.cs.berkeley.edu/data_overall.csv","source_label":"Official BFCL V4 data, Overall Acc column","reported_at":"","checked_at":"2026-09-16"},{"model":"Claude-Sonnet-4-5-20250929 (FC)","organization":"Anthropic","score":73.24,"metric":"Overall Acc (%)","setup":"BFCL V4; overall; FC.","reported_by":"leaderboard","source_url":"https://gorilla.cs.berkeley.edu/data_overall.csv","source_label":"Official BFCL V4 data, Overall Acc column","reported_at":"","checked_at":"2026-09-16"},{"model":"Gemini-3-Pro-Preview (Prompt)","organization":"Google","score":72.51,"metric":"Overall Acc (%)","setup":"BFCL V4; overall; Prompt.","reported_by":"leaderboard","source_url":"https://gorilla.cs.berkeley.edu/data_overall.csv","source_label":"Official BFCL V4 data, Overall Acc column","reported_at":"","checked_at":"2026-09-16"}]}},"facts":null},{"benchmark":{"slug":"bigcodebench","formal_name":"BigCodeBench","category":"Code","effective_mode":"full","profile":{"resources":[{"kind":"dataset","url":"https://huggingface.co/datasets/bigcode/bigcodebench","label":"Source dataset"},{"kind":"issues","url":"https://huggingface.co/datasets/bigcode/bigcodebench/discussions","label":"Dataset discussions"},{"kind":"site","url":"https://bigcode-bench.github.io/","label":"Official site"},{"kind":"paper","url":"https://arxiv.org/abs/2406.15877","label":"arXiv:2406.15877"}],"task_format":"Two forms of the same task — a complete docstring prompt and an instruction-style prompt — plus the code skeleton, the entry point, the libraries involved and the structured docstring.","scoring":"Correctness is decided by running a unittest suite against the candidate, so the reference solution the source ships is not a graded answer; it stays in restricted storage.","metric":"Pass@1 (the official runner also reports Pass@5 and Pass@10)","size":"1,139 unique problems imported from the default / v0.1.0_hf selection of 1,140 rows; 1 row repeats an earlier prompt and is recorded as a variant. Later version splits repeat the same task IDs and are not imported separately.","answer_access":"Columns outside the prompt allowlist, including any answer or reference solution, are separated into restricted storage and never rendered.","license":"apache-2.0 (declared on the Hugging Face dataset card)","history":"The BigCode project introduced BigCodeBench in 2024 to evaluate code generation that combines practical library functionality. It addressed a gap left by benchmarks dominated by short, self-contained functions. The dataset distribution recorded in this catalogue was published in June 2024.","construction":"BigCodeBench was developed through model-assisted task creation and human review of the resulting programming tasks. It contains 1,140 function-level tasks covering 139 libraries, with reference implementations and executable unit tests. The same tasks have a detailed docstring form and an instruction-style form, and correctness is evaluated by running candidate code against the tests.","limitations":"Passing the supplied tests establishes agreement with their checks rather than correctness for every possible input. Library versions and the execution environment matter because tasks depend on external APIs. Function-level generation does not directly assess maintaining a large repository or negotiating ambiguous requirements. Results from the docstring and instruction variants should be distinguished, as should single-attempt and multiple-attempt pass rates.","saturation":"","citation":"@article{zhuo2024bigcodebench,\n  title={BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions},\n  author={Terry Yue Zhuo and Minh Chien Vu and Jenny Chim and Han Hu and Wenhao Yu and Ratnadira Widyasari and Imam Nur Bani Yusuf and Haolan Zhan and Junda He and Indraneil Paul and Simon Brunner and Chen Gong and Thong Hoang and Armel Randy Zebaze and Xiaoheng Hong and Wen-Ding Li and Jean Kaddour and Ming Xu and Zhihan Zhang and Prateek Yadav and Naman Jain and Alex Gu and Zhoujun Cheng and Jiawei Liu and Qian Liu and Zijian Wang and Binyuan Hui and Niklas Muennighoff and David Lo and Daniel Fried and Xiaoning Du and Harm de Vries and Leandro Von Werra},\n  journal={arXiv preprint arXiv:2406.15877},\n  year={2024}\n}","maintainer":"bigcode (source organisation)","released":"Dataset published on Hugging Face 2024-06","why_hard":"The libraries have to be used as they actually behave, not as they are usually summarised. Most failures are correct-looking calls with the wrong arguments or the wrong return handling.","related":["humaneval-plus","mbpp-plus","ds-1000"],"scores":[]}},"facts":{"slug":"bigcodebench","task_count":1139,"image_count":0,"formats":[{"count":1139,"value":"code"}],"answers":[{"count":1139,"value":"published"}],"configs":[{"count":1139,"value":"default"}],"splits":[{"count":1139,"value":"v0.1.0_hf"}],"waiting_count":1139,"post_count":0,"discussed_count":0,"releases":[{"acquired":"2026-09-12","revision":"b74c0d0bf70d2c0bc459be537895cca163007f1a","release_id":"7fd659a6a4307945bef414ca0c4b3c1b4b8a37f7769ea03a1b5bedc1030fca3c"},{"acquired":"2026-09-12","revision":"b74c0d0bf70d2c0bc459be537895cca163007f1a","release_id":"4ef02375478c1b81a702f0787f529e0444e6e53ec8531cbd26f26ca4596dde16"},{"acquired":"2026-09-12","revision":"b74c0d0bf70d2c0bc459be537895cca163007f1a","release_id":"1b4e12bf295fb8678e573912219a672cd92b8c3f8dc287bee02585b4075720b8"}],"terms":{"license":null,"checked_at":null,"distribution_terms":null,"publisher_requests":null}}},{"benchmark":{"slug":"code-contests","formal_name":"CodeContests","category":"Code","effective_mode":"metadata_only","profile":{"resources":[{"kind":"dataset","url":"https://huggingface.co/datasets/deepmind/code_contests","label":"Source dataset"},{"kind":"issues","url":"https://huggingface.co/datasets/deepmind/code_contests/discussions","label":"Dataset discussions"},{"kind":"repository","url":"https://github.com/google-deepmind/code_contests","label":"Official repository"},{"kind":"paper","url":"https://arxiv.org/abs/2203.07814","label":"arXiv:2203.07814"}],"task_format":"A full competitive-programming statement with input and output specifications, time and memory limits.","scoring":"Correctness is decided by running the candidate against the public, private and generated tests.","metric":"","size":"Roughly 13,500 problems in the source; none imported.","answer_access":"No problem text is imported, so there is no answer storage for this entry. The source distributes the answers under its own terms.","license":"cc-by-4.0 (declared on the Hugging Face dataset card)","history":"DeepMind built CodeContests for its AlphaCode work in 2022 to study program generation on competitive-programming problems. The motivation was to evaluate whether a system could derive an algorithm from a problem statement and implement it under competition constraints. The catalogue records the source dataset publication on Hugging Face in July 2022.","construction":"The dataset collects existing competition problems from Codeforces and other programming-contest sources, together with correct and incorrect submissions. Problems include statements, input and output specifications, and execution limits. The source contains roughly 13,500 problems and combines public, private, and generated test cases for execution-based assessment.","limitations":"Contest programs emphasize self-contained algorithmic solutions rather than repository maintenance, requirements negotiation, or interaction with an existing software system. Passing a finite test suite does not prove correctness on every legal input. Candidate counts, selection procedures, execution environments, and time limits can change measured success, so results obtained with different generation and evaluation budgets are not directly interchangeable. Public contest statements and submissions also make training-data overlap possible.","saturation":"","citation":"@article{li2022codecontests,\n  title={Competition-Level Code Generation with AlphaCode},\n  author={Yujia Li and David Choi and Junyoung Chung and Nate Kushman and Julian Schrittwieser and Rémi Leblond and Tom Eccles and James Keeling and Felix Gimeno and Agustin Dal Lago and Thomas Hubert and Peter Choy and Cyprien de Masson d'Autume and Igor Babuschkin and Xinyun Chen and Po-Sen Huang and Johannes Welbl and Sven Gowal and Alexey Cherepanov and James Molloy and Daniel J. Mankowitz and Esme Sutherland Robson and Pushmeet Kohli and Nando de Freitas and Koray Kavukcuoglu and Oriol Vinyals},\n  journal={arXiv preprint arXiv:2203.07814},\n  year={2022}\n}","maintainer":"deepmind (source organisation)","released":"Dataset published on Hugging Face 2022-07","why_hard":"Competition problems hide an algorithmic idea behind a story, and a solution that is correct but too slow still fails the time limit.","related":["livecodebench","hlce","bigcodebench"],"scores":[]}},"facts":null},{"benchmark":{"slug":"humaneval","formal_name":"HumanEval","category":"Code","effective_mode":"full","profile":{"resources":[{"kind":"dataset","url":"https://huggingface.co/datasets/openai/openai_humaneval","label":"Source dataset"},{"kind":"issues","url":"https://huggingface.co/datasets/openai/openai_humaneval/discussions","label":"Dataset discussions"},{"kind":"paper","url":"https://arxiv.org/abs/2107.03374","label":"arXiv:2107.03374"}],"task_format":"A Python function signature and docstring in prompt, with the entry point named. The reference solution and the tests are kept out of the prompt.","scoring":"For the 164 imported problems the source ships a reference implementation and a unit-test suite; this catalogue keeps both in restricted storage and never renders them. Correctness is decided by executing a candidate against the tests, not by comparing it with the reference.","metric":"pass@1 (unbiased estimator over sampled completions)","size":"164 problems imported from the openai_humaneval / test selection, with a source-row limit of 200.","answer_access":"Columns outside the prompt allowlist, including any answer or reference solution, are separated into restricted storage and never rendered.","license":"mit (declared on the source)","history":"OpenAI introduced HumanEval in 2021 as part of its evaluation of Codex code-generation models. It made executable functional correctness a central measure of whether a model could complete a Python function from its specification. The March 2022 Hugging Face date recorded here refers to that distribution, not the original release.","construction":"HumanEval contains 164 manually written Python programming problems, each with a function signature, a docstring, a reference implementation and unit tests. A model supplies the function completion, and the evaluator runs it against the tests. Pass rates measure successful execution rather than textual similarity to the reference implementation.","limitations":"The small set of standalone Python functions does not measure repository navigation, dependency management or extended maintenance work. The original test suites can miss incorrect behaviour on untested inputs, so passing them is weaker than demonstrating general correctness. Public availability creates a risk of training exposure, while sampling settings and the number of allowed completions affect reported pass rates.","saturation":"The recorded vendor result for DeepSeek-V3 base is 65.2 percent pass@1 with no examples in the prompt, leaving headroom for that model and protocol. This single base-model result cannot establish overall saturation among specialised or instruction-tuned coding systems. With only 164 tasks, a small number of changed outcomes can noticeably move the aggregate score.","citation":"@misc{chen2021evaluating,\n      title={Evaluating Large Language Models Trained on Code},\n      author={Mark Chen and Jerry Tworek and Heewoo Jun and Qiming Yuan and Henrique Ponde de Oliveira Pinto and Jared Kaplan and Harri Edwards and Yuri Burda and Nicholas Joseph and Greg Brockman and Alex Ray and Raul Puri and Gretchen Krueger and Michael Petrov and Heidy Khlaaf and Girish Sastry and Pamela Mishkin and Brooke Chan and Scott Gray and Nick Ryder and Mikhail Pavlov and Alethea Power and Lukasz Kaiser and Mohammad Bavarian and Clemens Winter and Philippe Tillet and Felipe Petroski Such and Dave Cummings and Matthias Plappert and Fotios Chantzis and Elizabeth Barnes and Ariel Herbert-Voss and William Hebgen Guss and Alex Nichol and Alex Paino and Nikolas Tezak and Jie Tang and Igor Babuschkin and Suchir Balaji and Shantanu Jain and William Saunders and Christopher Hesse and Andrew N. Carr and Jan Leike and Josh Achiam and Vedant Misra and Evan Morikawa and Alec Radford and Matthew Knight and Miles Brundage and Mira Murati and Katie Mayer and Peter Welinder and Bob McGrew and Dario Amodei and Sam McCandlish and Ilya Sutskever and Wojciech Zaremba},\n      year={2021},\n      eprint={2107.03374},\n      archivePrefix={arXiv},\n      primaryClass={cs.LG}\n}","maintainer":"openai (source organisation)","released":"Dataset published on Hugging Face 2022-03","why_hard":"Correctness is decided by execution, not by resemblance to a reference: a plausible-looking function that fails one edge case scores zero.","related":["hlce","livecodebench","mbpp"],"scores":[{"model":"DeepSeek-V3","organization":"DeepSeek","score":65.2,"metric":"Pass@1","setup":"Base model; 0-shot.","reported_by":"vendor","source_url":"https://raw.githubusercontent.com/deepseek-ai/DeepSeek-V3/main/README.md","source_label":"Base Model, Standard Benchmarks, HumanEval (Pass@1), final column","reported_at":"","checked_at":"2026-09-16"}]}},"facts":{"slug":"humaneval","task_count":164,"image_count":0,"formats":[{"count":164,"value":"text"}],"answers":[{"count":164,"value":"published"}],"configs":[{"count":164,"value":"openai_humaneval"}],"splits":[{"count":164,"value":"test"}],"waiting_count":164,"post_count":0,"discussed_count":0,"releases":[{"acquired":"2026-09-11","revision":"7dce6050a7d6d172f3cc5c32aa97f52fa1a2e544","release_id":"29b40586e9cea9f0e93c06cbe954279d90875357e9d076ae54aa97f259d254ee"},{"acquired":"2026-09-11","revision":"7dce6050a7d6d172f3cc5c32aa97f52fa1a2e544","release_id":"09c39d799d2c00532ef542136cd4378dc049dbb61ff10c582f7bd491d22741f0"},{"acquired":"2026-09-11","revision":"7dce6050a7d6d172f3cc5c32aa97f52fa1a2e544","release_id":"143d6b426d5d2f8479b7fa164ec54a1e26b26f9edf8ea0dcac502a35ffe90990"},{"acquired":"2026-09-11","revision":"7dce6050a7d6d172f3cc5c32aa97f52fa1a2e544","release_id":"2bbe60b1260b1a116b16723b3b37c4c6d02874b07e35cf0a13a844b122129754"}],"terms":{"license":null,"checked_at":null,"distribution_terms":null,"publisher_requests":null}}},{"benchmark":{"slug":"humaneval-plus","formal_name":"HumanEval+","category":"Code","effective_mode":"full","profile":{"resources":[{"kind":"dataset","url":"https://huggingface.co/datasets/evalplus/humanevalplus","label":"Source dataset"},{"kind":"issues","url":"https://huggingface.co/datasets/evalplus/humanevalplus/discussions","label":"Dataset discussions"},{"kind":"repository","url":"https://github.com/evalplus/evalplus","label":"Official repository"},{"kind":"paper","url":"https://arxiv.org/abs/2305.01210","label":"arXiv:2305.01210"}],"task_format":"A Python function signature with its docstring, plus the entry-point name. The reference implementation and the test suite are not part of the prompt.","scoring":"Correctness is decided by executing the candidate against the expanded test suite, so the reference implementation the source ships is not a graded answer; it stays in restricted storage.","metric":"pass@1 on the expanded tests","size":"All 164 problems imported from the default / test selection, with a source-row limit of 200.","answer_access":"Columns outside the prompt allowlist, including any answer or reference solution, are separated into restricted storage and never rendered.","license":"apache-2.0 (declared on the Hugging Face dataset card)","history":"The EvalPlus research group introduced HumanEval+ in 2023 to investigate whether code-generation benchmarks were accepting incorrect programs because their tests were too weak. It retained the original HumanEval tasks while expanding the checks applied to candidate implementations. The January 2024 publication date recorded here refers to the evalplus Hugging Face distribution.","construction":"HumanEval+ retains all 164 Python function-completion problems from HumanEval and expands their tests by roughly a factor of 80. The EvalPlus approach combines model-generated test inputs with mutation-based generation to explore additional cases, using reference implementations to obtain expected outputs for valid inputs. A candidate is evaluated by executing its function against the expanded test suite, rather than by comparing its source text with a reference solution.","limitations":"Expanded tests improve detection of incorrect implementations but do not prove correctness for every possible input. Evaluation still depends on the adequacy of task specifications, reference implementations and the particular test-suite version. The tasks are short, standalone Python functions and do not measure repository navigation, maintenance or integration with a larger software system. Reusing public HumanEval prompts also leaves potential prior exposure unresolved.","saturation":"","citation":"@article{liu2023humanevalplus,\n  title={Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation},\n  author={Jiawei Liu and Chunqiu Steven Xia and Yuyao Wang and Lingming Zhang},\n  journal={arXiv preprint arXiv:2305.01210},\n  year={2023}\n}","maintainer":"evalplus (source organisation)","released":"Dataset published on Hugging Face 2024-01","why_hard":"The problems are short, but the added tests probe the edge cases the original suite missed, so a solution that only handles the obvious path now fails.","related":["humaneval","mbpp-plus","bigcodebench"],"scores":[]}},"facts":{"slug":"humaneval-plus","task_count":164,"image_count":0,"formats":[{"count":164,"value":"code"}],"answers":[{"count":164,"value":"published"}],"configs":[{"count":164,"value":"default"}],"splits":[{"count":164,"value":"test"}],"waiting_count":164,"post_count":0,"discussed_count":0,"releases":[{"acquired":"2026-09-12","revision":"d32357cf319e50e9c8d8dab5ea876c72b0fd321b","release_id":"aa57f7510089d75496a9e14e1beff9ddadcc28fde32c55b163e3ffb36c75de99"},{"acquired":"2026-09-12","revision":"d32357cf319e50e9c8d8dab5ea876c72b0fd321b","release_id":"8cda5f6854a1a383f3ae78c9194dc11fa557660776b108db43edab5c95e7cc05"},{"acquired":"2026-09-12","revision":"d32357cf319e50e9c8d8dab5ea876c72b0fd321b","release_id":"ad959cd691f55fa75b68008286fc08b5528108aeed133b5fd3af3fabcb987378"}],"terms":{"license":null,"checked_at":null,"distribution_terms":null,"publisher_requests":null}}},{"benchmark":{"slug":"hlce","formal_name":"Humanity's Last Code Exam","category":"Code","effective_mode":"full","profile":{"resources":[{"kind":"repository","url":"https://github.com/Humanity-s-Last-Code-Exam/HLCE","label":"Source repository"},{"kind":"issues","url":"https://github.com/Humanity-s-Last-Code-Exam/HLCE/issues","label":"Repository issues"},{"kind":"site","url":"https://humanity-s-last-code-exam.github.io/website/","label":"Official site"}],"task_format":"Competitive-programming problem statements; the import accepts question_title, question_content, and platform.","scoring":"For the 146 imported problems, the source ships an answer with every problem; this catalogue keeps it in restricted storage and never renders it. Definitive results come from the IOI and ICPC World Finals scripts in the official repository.","metric":"","size":"235 problems described in the catalog introduction; this selection expects 146 ICPC problem records from ICPC-World-Finals / examples, with a source-row limit of 200.","answer_access":"Generated responses and scoring outputs in the selected example data remain in restricted storage.","license":"","history":"The Humanity's Last Code Exam project assembled a benchmark of ICPC World Finals and International Olympiad in Informatics problems to evaluate demanding algorithmic problem solving and code generation. Its GitHub repository, maintained by the Humanity-s-Last-Code-Exam organisation, was created in May 2025. The project drew on established programming competitions as a source of difficult tasks.","construction":"The project describes 235 problems drawn from ICPC World Finals and IOI competitions. This catalogue selects 146 public ICPC problem records, whose prompt fields include the title, statement, and platform. Evaluation is defined by the official repository scripts for the two competitions; generated responses and scoring outputs in the selected example data are not themselves independent correctness criteria.","limitations":"The imported ICPC subset does not cover the full benchmark or establish performance on the IOI portion. Competitive-programming tasks test algorithm design and implementation under contest conditions, with limited coverage of software maintenance or collaborative development. Because the problems originate in public competitions, exposure to statements or solutions during training is a possibility that requires separate investigation.","saturation":"","citation":"@misc{li2025humanityscodeexamadvanced,\n      title={Humanity's Last Code Exam: Can Advanced LLMs Conquer Human's Hardest Code Competition?}, \n      author={Xiangyang Li and Xiaopeng Li and Kuicai Dong and Quanhu Zhang and Rongju Ruan and Xinyi Dai and Xiaoshuang Liu and Shengchun Xu and Yasheng Wang and Ruiming Tang},\n      year={2025},\n      eprint={2506.12713},\n      archivePrefix={arXiv},\n      primaryClass={cs.SE},\n      url={https://arxiv.org/abs/2506.12713}, \n}","maintainer":"Humanity-s-Last-Code-Exam (GitHub organisation)","released":"Repository created on GitHub 2025-05","why_hard":"Requires problem solving and code generation on ICPC World Finals and IOI problems.","related":["scicode","swe-bench-pro","terminal-bench-2-1"],"scores":[]}},"facts":{"slug":"hlce","task_count":146,"image_count":0,"formats":[{"count":146,"value":"text"}],"answers":[{"count":146,"value":"published"}],"configs":[{"count":146,"value":"ICPC-World-Finals"}],"splits":[{"count":146,"value":"examples"}],"waiting_count":146,"post_count":0,"discussed_count":0,"releases":[{"acquired":"2026-09-09","revision":"7ff35699eb56a09ba17ea00ed3dbb6e520635832","release_id":"aa405bca77f4c059fd7a45c7b65c691035174d7f20563230b84650fd28aa3d5c"},{"acquired":"2026-09-09","revision":"7ff35699eb56a09ba17ea00ed3dbb6e520635832","release_id":"ce11e5bbbba75a64bd935da4b732ee215e53e2901d59f3f3ad164123b0ce3365"},{"acquired":"2026-09-09","revision":"7ff35699eb56a09ba17ea00ed3dbb6e520635832","release_id":"31d1c4183bd060e29fed08a392edd151f814777173be1ee9e121c9f263c627e9"},{"acquired":"2026-09-09","revision":"7ff35699eb56a09ba17ea00ed3dbb6e520635832","release_id":"20b0aecfd177c0d5df6caee3b549302b769ea05c24917abe44c8264b19e39763"}],"terms":{"license":null,"checked_at":null,"distribution_terms":null,"publisher_requests":null}}},{"benchmark":{"slug":"livecodebench","formal_name":"LiveCodeBench","category":"Code","effective_mode":"full","profile":{"resources":[{"kind":"dataset","url":"https://huggingface.co/datasets/livecodebench/code_generation_lite","label":"Source dataset"},{"kind":"issues","url":"https://huggingface.co/datasets/livecodebench/code_generation_lite/discussions","label":"Dataset discussions"},{"kind":"site","url":"https://livecodebench.github.io/","label":"Official site"},{"kind":"paper","url":"https://arxiv.org/abs/2403.07974","label":"arXiv:2403.07974"}],"task_format":"Competitive-programming questions; the import accepts question_title, question_content, platform, contest_id, contest_date, difficulty, and starter_code.","scoring":"Generated programs are evaluated with public_test_cases and private_test_cases using an external runner.","metric":"","size":"400 problems in initial release_v1 described in the catalog introduction; this release_v1 / test selection expects 100 task records, limited to the first 100 source rows.","answer_access":"Test cases remain outside the prompt in restricted storage. They support execution-based evaluation and are not reference programs.","license":"cc (declared on the Hugging Face dataset card)","history":"The LiveCodeBench research group introduced the benchmark in 2024 to evaluate coding capabilities using newly collected competitive-programming problems. Continuing to add problems made it possible to evaluate on material released after a model training cutoff, addressing a weakness of static public coding datasets. The catalogue records publication of the source dataset on Hugging Face in April 2024.","construction":"LiveCodeBench collects competition questions and records their source platform, contest date, difficulty, and dataset version. Generated programs are checked against public and private test cases using an external execution runner. The initial release_v1 contains 400 problems, while this catalogue selection includes only the first 100 source records from that release.","limitations":"A benchmark name alone does not identify a LiveCodeBench evaluation: release version and contest-date window determine which problems are included. Recent problems reduce the opportunity for training exposure only when their dates are later than the relevant training cutoff; an older public release does not retain that advantage indefinitely. Execution settings and sampling budgets also affect comparisons, and competition code covers only part of practical software engineering.","saturation":"","citation":"","maintainer":"livecodebench (Hugging Face organisation)","released":"Dataset published on Hugging Face 2024-04","why_hard":"Requires solving competitive-programming problems; the dataset continuously collects new problems.","related":["hlce","scicode","swe-bench-pro"],"scores":[]}},"facts":{"slug":"livecodebench","task_count":100,"image_count":0,"formats":[{"count":100,"value":"text"}],"answers":[{"count":100,"value":"external_grader"}],"configs":[{"count":100,"value":"release_v1"}],"splits":[{"count":100,"value":"test"}],"waiting_count":100,"post_count":0,"discussed_count":0,"releases":[{"acquired":"2026-09-09","revision":"0fe84c3912ea0c4d4a78037083943e8f0c4dd505","release_id":"93a2ce37da599419f0e370d41204b1bcd17a7b11dd4c98bde6eb4223da08701a"},{"acquired":"2026-09-09","revision":"0fe84c3912ea0c4d4a78037083943e8f0c4dd505","release_id":"f75d3beb07eace217c7bf9636175ca4a529d29298b4399b5111473917e8ddae8"},{"acquired":"2026-09-09","revision":"0fe84c3912ea0c4d4a78037083943e8f0c4dd505","release_id":"2167c113d1ed6b74bbbbdbef9ff5e67366d4052476ca048b6a382095dfdc9fef"},{"acquired":"2026-09-09","revision":"0fe84c3912ea0c4d4a78037083943e8f0c4dd505","release_id":"8a40ca236bd7d956f33d833b832dcb2605dfe362772813881864f426da79939e"}],"terms":{"license":null,"checked_at":null,"distribution_terms":null,"publisher_requests":null}}},{"benchmark":{"slug":"mbpp","formal_name":"MBPP","category":"Code","effective_mode":"full","profile":{"resources":[{"kind":"dataset","url":"https://huggingface.co/datasets/google-research-datasets/mbpp","label":"Source dataset"},{"kind":"issues","url":"https://huggingface.co/datasets/google-research-datasets/mbpp/discussions","label":"Dataset discussions"},{"kind":"paper","url":"https://arxiv.org/abs/2108.07732","label":"arXiv:2108.07732"}],"task_format":"A one-sentence task description (text in the full config, prompt in sanitized) with the assertion list that names the expected function. The reference code is kept out of the prompt.","scoring":"The source ships a reference program with every problem; it stays in restricted storage and is never rendered. The assertion tests are part of the problem statement and are shown with it; correctness is decided by executing a candidate against the assertions, not by comparing it with the reference.","metric":"pass@1 on the assertion tests","size":"1,401 problems imported: the full config across its four splits (974 records) and the hand-verified sanitized config across its four splits (427 records). The two configs overlap by upstream task_id and are kept as separate variant pages.","answer_access":"Columns outside the prompt allowlist, including any answer or reference solution, are separated into restricted storage and never rendered.","license":"cc-by-4.0 (declared on the source)","history":"Google researchers introduced Mostly Basic Python Problems in 2021 to study program synthesis from natural-language descriptions. The benchmark focused on short Python tasks suitable for checking with executable tests. The Google-hosted dataset recorded in this catalog was published on Hugging Face in March 2022, after the original research release.","construction":"MBPP was assembled from crowd-sourced Python programming tasks, each accompanied by a natural-language description, reference code and assertion tests. The sanitized version is a hand-verified subset intended to improve task quality. This catalog imports 200 problems from the sanitized test selection and keeps both reference programs and assertions out of the displayed prompt. Candidate programs are judged by executing the tests rather than matching the reference text.","limitations":"A short description can leave behavior or the expected function interface ambiguous, particularly when tests are withheld from the prompt. Passing a finite set of assertions does not establish correctness on untested inputs, and small standalone functions do not measure repository-scale software engineering. Comparisons also depend on whether they use the original or sanitized dataset and whether demonstrations or tests are supplied. Public availability creates a potential training-contamination route without establishing exposure for any particular model.","saturation":"The vendor result recorded here reports DeepSeek-V3 at 75.4 pass@1 in a base-model, three-shot setup. That report leaves headroom under its stated conditions, but it does not identify the present ceiling of the benchmark. It should not be read as a result specifically on this catalog's 200 imported sanitized tasks unless the evaluation subset is independently confirmed.","citation":"@article{austin2021program,\n  title={Program Synthesis with Large Language Models},\n  author={Austin, Jacob and Odena, Augustus and Nye, Maxwell and Bosma, Maarten and Michalewski, Henryk and Dohan, David and Jiang, Ellen and Cai, Carrie and Terry, Michael and Le, Quoc and others},\n  journal={arXiv preprint arXiv:2108.07732},\n  year={2021}","maintainer":"google-research-datasets (source organisation)","released":"Dataset published on Hugging Face 2022-03","why_hard":"The description is short enough to be ambiguous, so the model has to infer the intended signature and behaviour from the tests it cannot see.","related":["hlce","humaneval","livecodebench"],"scores":[{"model":"DeepSeek-V3","organization":"DeepSeek","score":75.4,"metric":"Pass@1","setup":"Base model; 3-shot.","reported_by":"vendor","source_url":"https://raw.githubusercontent.com/deepseek-ai/DeepSeek-V3/main/README.md","source_label":"Base Model, Standard Benchmarks, MBPP (Pass@1), final column","reported_at":"","checked_at":"2026-09-16"}]}},"facts":{"slug":"mbpp","task_count":1401,"image_count":0,"formats":[{"count":1401,"value":"text"}],"answers":[{"count":1401,"value":"published"}],"configs":[{"count":974,"value":"full"},{"count":427,"value":"sanitized"}],"splits":[{"count":17,"value":"prompt"},{"count":757,"value":"test"},{"count":494,"value":"train"},{"count":133,"value":"validation"}],"waiting_count":1401,"post_count":0,"discussed_count":0,"releases":[{"acquired":"2026-09-23","revision":"4bb6404fdc6cacfda99d4ac4205087b89d32030c","release_id":"f1af1b73310cf6b71e07ee109f4ca2bf623e1a46dad5664525d7bf76d12a53b8"},{"acquired":"2026-09-11","revision":"4bb6404fdc6cacfda99d4ac4205087b89d32030c","release_id":"a9c6c874725554fdb086297636d574e24552c6ccacd74801e10bcd4feb08b2b2"},{"acquired":"2026-09-11","revision":"4bb6404fdc6cacfda99d4ac4205087b89d32030c","release_id":"ecd40ec0a69dd82ee51b3fb6a00e885dde5b02802b7d1006b67d02c6aa54bcee"},{"acquired":"2026-09-11","revision":"4bb6404fdc6cacfda99d4ac4205087b89d32030c","release_id":"67cf16715b15f43631b59929eef220cc2a0ab79e0e2ba167b68d5496b37895b0"}],"terms":{"license":null,"checked_at":null,"distribution_terms":null,"publisher_requests":null}}},{"benchmark":{"slug":"mbpp-plus","formal_name":"MBPP+","category":"Code","effective_mode":"full","profile":{"resources":[{"kind":"dataset","url":"https://huggingface.co/datasets/evalplus/mbppplus","label":"Source dataset"},{"kind":"issues","url":"https://huggingface.co/datasets/evalplus/mbppplus/discussions","label":"Dataset discussions"},{"kind":"repository","url":"https://github.com/evalplus/evalplus","label":"Official repository"},{"kind":"paper","url":"https://arxiv.org/abs/2305.01210","label":"arXiv:2305.01210"}],"task_format":"A short natural-language programming task. The reference program and the expanded assertions are not part of the prompt.","scoring":"Correctness is decided by executing the candidate against the expanded assertions, so the reference program the source ships is not a graded answer; it stays in restricted storage.","metric":"pass@1 on the expanded tests","size":"378 problems imported from the default / test selection (the full split).","answer_access":"Columns outside the prompt allowlist, including any answer or reference solution, are separated into restricted storage and never rendered.","license":"apache-2.0 (declared on the Hugging Face dataset card)","history":"The EvalPlus project developed MBPP+ to strengthen the execution-based evaluation of programs on MBPP. Its Hugging Face distribution was published in January 2024. The motivation was to catch incorrect programs that pass a small original test suite, a problem also addressed by HumanEval+.","construction":"MBPP+ retains the 378 programming problems of the sanitized MBPP set and expands their tests by roughly 35 times. The task remains a short natural-language programming request, with candidate programs evaluated by running the expanded assertions. This catalogue imports 200 of the 378 problems and keeps reference programs and expanded assertions outside the displayed prompt.","limitations":"Additional tests provide stronger evidence of correctness but cannot prove that a program satisfies a specification for every possible input. Short task descriptions can also leave edge cases underspecified, making the reference implementation and tests consequential parts of the evaluation. The benchmark covers small programming tasks rather than repository-scale development, and expanding tests does not remove potential exposure to the original public problems.","saturation":"","citation":"@article{liu2023mbppplus,\n  title={Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation},\n  author={Jiawei Liu and Chunqiu Steven Xia and Yuyao Wang and Lingming Zhang},\n  journal={arXiv preprint arXiv:2305.01210},\n  year={2023}\n}","maintainer":"evalplus (source organisation)","released":"Dataset published on Hugging Face 2024-01","why_hard":"Each task is a few lines of Python, but the specification is one sentence long, so most of the difficulty is in reading what is actually being asked for.","related":["mbpp","humaneval-plus","bigcodebench"],"scores":[]}},"facts":{"slug":"mbpp-plus","task_count":378,"image_count":0,"formats":[{"count":378,"value":"code"}],"answers":[{"count":378,"value":"published"}],"configs":[{"count":378,"value":"default"}],"splits":[{"count":378,"value":"test"}],"waiting_count":378,"post_count":0,"discussed_count":0,"releases":[{"acquired":"2026-09-12","revision":"b2d74c91837c3f2a20c1299ae98133cbe7cfa077","release_id":"10a7a2abecadd00085700769ebc3f6b1fdf4428c85da57b1b98bb937dbe5f75f"},{"acquired":"2026-09-12","revision":"b2d74c91837c3f2a20c1299ae98133cbe7cfa077","release_id":"532e203491c386b129fdc74d4c22501514983d9e55e7f940606afcbc08feb774"},{"acquired":"2026-09-12","revision":"b2d74c91837c3f2a20c1299ae98133cbe7cfa077","release_id":"147b31500dd9f9fdd1a5e0c27388dd23c39201af4481cd784df14fd32076f9ab"}],"terms":{"license":null,"checked_at":null,"distribution_terms":null,"publisher_requests":null}}},{"benchmark":{"slug":"swe-bench-pro","formal_name":"SWE-Bench Pro","category":"Code / Agents","effective_mode":"full","profile":{"resources":[{"kind":"dataset","url":"https://huggingface.co/datasets/ScaleAI/SWE-bench_Pro","label":"Source dataset"},{"kind":"issues","url":"https://huggingface.co/datasets/ScaleAI/SWE-bench_Pro/discussions","label":"Dataset discussions"}],"task_format":"Repository repair tasks; the import accepts problem_statement, requirements, interface, repo, base_commit, repo_language, and dockerhub_tag.","scoring":"Repository patches are evaluated with fail_to_pass and pass_to_pass tests; a patch alone is not a graded result.","metric":"","size":"731 public tasks described in the catalog introduction; this test selection expects 200 task records with a source-row limit of 200.","answer_access":"Patches and tests are kept outside prompt fields in restricted storage; evaluation requires a test runner, not just a reference patch.","license":"","history":"Scale AI distributes SWE-Bench Pro as an evaluation of agents performing extended software engineering work in real repositories. This catalogue records publication of the Hugging Face dataset in August 2025. Its focus is repository repair over a sequence of actions, rather than isolated function completion.","construction":"The public dataset card describes 731 tasks with issue descriptions, repository identifiers and base commits. Task records also specify requirements and interfaces, placing each requested repair in a particular repository state. Evaluation uses tests that must change from failing to passing and tests that must continue to pass; matching a reference patch is not the scoring criterion.","limitations":"Test success measures the behaviours exercised by the evaluation suite rather than every consequence of a repository change. Agent scaffolding, available tools, runtime environment and execution budget can affect the result as well as the underlying model. The 731 tasks described in the public card should be identified as that public collection when comparing reports, rather than assumed to represent every evaluation carrying the SWE-Bench Pro name.","saturation":"","citation":"","maintainer":"ScaleAI (Hugging Face organisation)","released":"Dataset published on Hugging Face 2025-08","why_hard":"Requires long-horizon software engineering in real repositories.","related":["terminal-bench-2-1","hlce","scicode"],"scores":[]}},"facts":{"slug":"swe-bench-pro","task_count":200,"image_count":0,"formats":[{"count":200,"value":"text"}],"answers":[{"count":200,"value":"external_grader"}],"configs":[{"count":200,"value":""}],"splits":[{"count":200,"value":"test"}],"waiting_count":200,"post_count":0,"discussed_count":0,"releases":[{"acquired":"2026-09-09","revision":"7ab5114912baf22bb098818e604c02fe7ad2c11f","release_id":"4cd4b68249abfc826b48bf27cd48958ae2b1c291767ed2ddbc9ec55ed3f9ce22"},{"acquired":"2026-09-09","revision":"7ab5114912baf22bb098818e604c02fe7ad2c11f","release_id":"16026b211da6d99c7d956e66794ed591c74b87a24abf50739dcacb6a3e698737"},{"acquired":"2026-09-09","revision":"7ab5114912baf22bb098818e604c02fe7ad2c11f","release_id":"afc4a41520ba95de4119bd4cdaa6e02b97e496f04bd4188946e83099ccaf1b78"},{"acquired":"2026-09-09","revision":"7ab5114912baf22bb098818e604c02fe7ad2c11f","release_id":"43e81a1c7541f44c792928a5142eaa17ef03b00e3de5b30d3ec2b9b63d4946fc"}],"terms":{"license":null,"checked_at":null,"distribution_terms":null,"publisher_requests":null}}},{"benchmark":{"slug":"swe-bench-lite","formal_name":"SWE-bench Lite","category":"Code / Agents","effective_mode":"metadata_only","profile":{"resources":[{"kind":"dataset","url":"https://huggingface.co/datasets/princeton-nlp/SWE-bench_Lite","label":"Source dataset"},{"kind":"issues","url":"https://huggingface.co/datasets/princeton-nlp/SWE-bench_Lite/discussions","label":"Dataset discussions"},{"kind":"site","url":"https://www.swebench.com/","label":"Official site"},{"kind":"paper","url":"https://arxiv.org/abs/2310.06770","label":"arXiv:2310.06770"}],"task_format":"An issue statement plus the repository and base commit it applies to.","scoring":"Correctness is decided by running the repository’s own fail-to-pass and pass-to-pass tests against the candidate patch.","metric":"","size":"300 instances in the source; none imported.","answer_access":"No problem text is imported, so there is no answer storage for this entry. The source distributes the answers under its own terms.","license":"Not declared on the source.","history":"The SWE-bench team published SWE-bench Lite through Princeton NLP as a smaller evaluation set for repository-level software repair, and this catalogue records its Hugging Face publication as March 2024 rather than a separate announcement date. It retains the central task of producing a patch for a real GitHub issue while reducing the cost of repeated evaluation. The subset contains 300 instances.","construction":"SWE-bench Lite is selected from SWE-bench, whose instances link real GitHub issues to the repository changes that resolved them. Each task supplies an issue statement and a repository base commit from which a candidate patch is developed. Evaluation runs repository tests, including tests expected to change from failing to passing and tests expected to remain passing. The 300-instance subset therefore checks patches in the context of existing projects rather than as isolated code snippets.","limitations":"Passing the selected tests establishes success under the harness, not that a patch satisfies every unstated requirement or avoids every possible regression. Results reflect the agent's code navigation, editing and execution tools as well as its underlying model. Public issues and historical fixes also create a potential training-data overlap risk. Lite is a particular subset, so its resolution rate should not be substituted for a result on full SWE-bench or SWE-bench Verified.","saturation":"The recorded official leaderboard reports that SWE-agent with Claude 4 Sonnet resolved 56.67% of Lite instances in a single-attempt setup on May 26, 2025. A February 26, 2025 entry for SWE-agent with Claude 3.7 Sonnet reports 48.0%. These entries leave substantial headroom, but they describe particular agent-model combinations and do not establish the best attainable rate under larger attempt budgets.","citation":"@article{jimenez2023swebenchlite,\n  title={SWE-bench: Can Language Models Resolve Real-World GitHub Issues?},\n  author={Carlos E. Jimenez and John Yang and Alexander Wettig and Shunyu Yao and Kexin Pei and Ofir Press and Karthik Narasimhan},\n  journal={arXiv preprint arXiv:2310.06770},\n  year={2023}\n}","maintainer":"princeton-nlp (source organisation)","released":"Dataset published on Hugging Face 2024-03","why_hard":"The fix has to work inside a real codebase at a real commit, so the model has to locate the relevant code before it can change it.","related":["swe-bench-verified","swe-bench-pro","terminal-bench-2-1"],"scores":[{"model":"SWE-agent + Claude 4 Sonnet","organization":"Anthropic","score":56.67,"metric":"resolved (%)","setup":"Lite; SWE-agent; Model: claude-4-sonnet-20250514; Org: SWE-agent; System: Attempts - 1","reported_by":"leaderboard","source_url":"https://raw.githubusercontent.com/SWE-bench/swe-bench.github.io/master/data/leaderboards.json","source_label":"Lite leaderboard; 20250526_sweagent_claude-4-sonnet-20250514","reported_at":"2025-05-26","checked_at":"2026-09-16"},{"model":"SWE-agent + Claude 3.7 Sonnet","organization":"Anthropic","score":48,"metric":"resolved (%)","setup":"Lite; SWE-agent; Model: claude-3-7-sonnet-20250219; Org: SWE-agent; System: Attempts - 1","reported_by":"leaderboard","source_url":"https://raw.githubusercontent.com/SWE-bench/swe-bench.github.io/master/data/leaderboards.json","source_label":"Lite leaderboard; 20250226_sweagent_claude-3-7-sonnet-20250219","reported_at":"2025-02-26","checked_at":"2026-09-16"}]}},"facts":null},{"benchmark":{"slug":"swe-bench-verified","formal_name":"SWE-bench Verified","category":"Code / Agents","effective_mode":"metadata_only","profile":{"resources":[{"kind":"dataset","url":"https://huggingface.co/datasets/princeton-nlp/SWE-bench_Verified","label":"Source dataset"},{"kind":"issues","url":"https://huggingface.co/datasets/princeton-nlp/SWE-bench_Verified/discussions","label":"Dataset discussions"}],"task_format":"A real GitHub issue with the repository, base commit and the tests that must go from failing to passing.","scoring":"The dataset declares no licence, and a run needs the repository checkout and test harness rather than the issue text alone. The catalogue links to the official distribution.","metric":"","size":"Not imported here; see the official distribution for the current size.","answer_access":"The dataset declares no licence, and a run needs the repository checkout and test harness rather than the issue text alone. The catalogue links to the official distribution.","license":"","history":"SWE-bench Verified was released in August 2024 through work by OpenAI and the SWE-bench team to improve the reliability of repository-level software engineering evaluation. It selected tasks from the original SWE-bench collection after human review of whether their issue descriptions and tests supported a fair, solvable task. The dataset is hosted under princeton-nlp on Hugging Face.","construction":"The benchmark contains 500 tasks drawn from real GitHub issues in Python repositories. Each task is tied to a base commit and a test-based evaluation of a candidate patch. Human annotators screened the tasks for solvability and specification quality, reducing cases where an unclear request or unsuitable tests could confound the result. Successful patches must fix the targeted failing tests while preserving the required previously passing tests.","limitations":"A resolved task means that the patch satisfies the evaluation tests, not that it meets every unstated user expectation or is ready for production. The collection covers a particular set of public Python repositories, so it does not represent all languages or software engineering work. Public issues, historical patches and test data create potential prior-exposure risks. Agent tools, retry budgets and scaffolding also affect results, making the reported score a property of the complete evaluation system.","saturation":"The leaderboard records included here report 74.4 percent resolved for Claude 4.5 Opus at medium effort and 74.2 percent for Gemini 3 Pro Preview, using mini-SWE-agent in November 2025. These results leave roughly a quarter of the tasks unresolved in the reported configurations. The records use different mini-SWE-agent versions, so even their close scores should not be read as a controlled comparison of models alone. They demonstrate remaining headroom in those reports rather than establishing the current leaderboard ceiling.","citation":"","maintainer":"princeton-nlp (source organisation)","released":"Dataset published on Hugging Face 2024-08","why_hard":"A patch must make specific tests pass without breaking others, inside a real codebase the model has not seen.","related":["swe-bench-pro","terminal-bench-2-1","arc-agi-3"],"scores":[{"model":"Claude 4.5 Opus (20251101) (medium)","organization":"Anthropic","score":74.4,"metric":"resolved (%)","setup":"Verified; mini-SWE-agent; Model: claude-opus-4-5-20251101; Org: Anthropic; System: Attempts - 1; Mini: 1.16.0","reported_by":"leaderboard","source_url":"https://raw.githubusercontent.com/SWE-bench/swe-bench.github.io/master/data/leaderboards.json","source_label":"Verified leaderboard; 20251124_mini-v1.16.0_claude-opus-4-5-20251101","reported_at":"2025-11-24","checked_at":"2026-09-16"},{"model":"Gemini 3 Pro Preview (2025-11-18)","organization":"Google DeepMind","score":74.2,"metric":"resolved (%)","setup":"Verified; mini-SWE-agent; Model: gemini-3-pro-preview; Org: Google DeepMind; System: Attempts - 1; Mini: 1.15.0","reported_by":"leaderboard","source_url":"https://raw.githubusercontent.com/SWE-bench/swe-bench.github.io/master/data/leaderboards.json","source_label":"Verified leaderboard; 20251118_mini-v1.15.0_gemini-3-pro-preview-20251118","reported_at":"2025-11-18","checked_at":"2026-09-16"},{"model":"OpenHands + GPT-5","organization":"OpenAI","score":71.8,"metric":"resolved (%)","setup":"Verified; OpenHands; Model: openai/gpt-5-2025-08-07; Org: OpenHands; System: Attempts - 1","reported_by":"leaderboard","source_url":"https://raw.githubusercontent.com/SWE-bench/swe-bench.github.io/master/data/leaderboards.json","source_label":"Verified leaderboard; 20250807_openhands_gpt5","reported_at":"2025-08-07","checked_at":"2026-09-16"}]}},"facts":null},{"benchmark":{"slug":"cruxeval","formal_name":"CRUXEval","category":"Code / Reasoning","effective_mode":"full","profile":{"resources":[{"kind":"dataset","url":"https://huggingface.co/datasets/cruxeval-org/cruxeval","label":"Source dataset"},{"kind":"issues","url":"https://huggingface.co/datasets/cruxeval-org/cruxeval/discussions","label":"Dataset discussions"},{"kind":"site","url":"https://crux-eval.github.io/","label":"Official site"},{"kind":"paper","url":"https://arxiv.org/abs/2401.03065","label":"arXiv:2401.03065"}],"task_format":"A short Python function and one input. The expected output is not part of the prompt.","scoring":"The source ships the expected output with every problem; this catalogue keeps it in restricted storage and never renders it.","metric":"pass@1 on output prediction and on input prediction, reported separately","size":"800 problems imported from the default / test selection (the full split).","answer_access":"Columns outside the prompt allowlist, including any answer or reference solution, are separated into restricted storage and never rendered.","license":"mit (declared on the Hugging Face dataset card)","history":"The CRUXEval research group introduced the benchmark in a 2024 paper to separate code reasoning from code generation. Its two tasks ask a model either to predict a function's output or to find an input that yields a specified output. This catalogue records a November 2023 Hugging Face publication, preceding the paper.","construction":"CRUXEval consists of 800 short Python functions with associated inputs and outputs. The functions were generated with a code model and filtered, with execution used to establish input-output consistency. Each function supports output prediction and input prediction, which are evaluated separately.","limitations":"Short, synthetic Python functions provide a narrow test of program behaviour rather than a measure of repository-level software engineering. Input prediction may admit multiple correct inputs, so correctness concerns the output produced rather than matching one reference input. Access to a Python interpreter changes the task substantially and needs to be distinguished from reasoning without execution.","saturation":"The recorded vendor results for DeepSeek-V3 base are 69.8 percent on output prediction and 67.3 percent on input prediction, both with two examples in the prompt. These reports leave substantial headroom in each direction under that setup. They do not establish the present best result across all systems or tool-use protocols.","citation":"@article{gu2024cruxeval,\n  title={CRUXEval: A Benchmark for Code Reasoning, Understanding and Execution},\n  author={Alex Gu and Baptiste Rozière and Hugh Leather and Armando Solar-Lezama and Gabriel Synnaeve and Sida I. Wang},\n  journal={arXiv preprint arXiv:2401.03065},\n  year={2024}\n}","maintainer":"cruxeval-org (source organisation)","released":"Dataset published on Hugging Face 2023-11","why_hard":"Writing the function is not the task; simulating it is. Models that generate correct code often still mispredict what their own code returns.","related":["humaneval-plus","bigcodebench","mbpp-plus"],"scores":[{"model":"DeepSeek-V3","organization":"DeepSeek","score":69.8,"metric":"Acc.","setup":"Base model; 2-shot; output prediction.","reported_by":"vendor","source_url":"https://raw.githubusercontent.com/deepseek-ai/DeepSeek-V3/main/README.md","source_label":"Base Model, Standard Benchmarks, CRUXEval-O (Acc.), final column","reported_at":"","checked_at":"2026-09-16"},{"model":"DeepSeek-V3","organization":"DeepSeek","score":67.3,"metric":"Acc.","setup":"Base model; 2-shot; input prediction.","reported_by":"vendor","source_url":"https://raw.githubusercontent.com/deepseek-ai/DeepSeek-V3/main/README.md","source_label":"Base Model, Standard Benchmarks, CRUXEval-I (Acc.), final column","reported_at":"","checked_at":"2026-09-16"}]}},"facts":{"slug":"cruxeval","task_count":800,"image_count":0,"formats":[{"count":800,"value":"code"}],"answers":[{"count":800,"value":"published"}],"configs":[{"count":800,"value":"default"}],"splits":[{"count":800,"value":"test"}],"waiting_count":800,"post_count":0,"discussed_count":0,"releases":[{"acquired":"2026-09-12","revision":"b96af0450242eb4da433032b90998f25588a5d0f","release_id":"95e7b4d7414774b5bae2fee3271821bed8d02db9da05844e9ffdf249c66757b4"},{"acquired":"2026-09-12","revision":"b96af0450242eb4da433032b90998f25588a5d0f","release_id":"292153319d1f577b71ff7188ee1647c192a81ef119e72d2ab517f9329cedd871"},{"acquired":"2026-09-12","revision":"b96af0450242eb4da433032b90998f25588a5d0f","release_id":"a5abcc0b205a71f6f7f8d79cfa334f497d7f7829ce48fc1960192e66fd4857af"}],"terms":{"license":null,"checked_at":null,"distribution_terms":null,"publisher_requests":null}}},{"benchmark":{"slug":"ds-1000","formal_name":"DS-1000","category":"Code / Science","effective_mode":"full","profile":{"resources":[{"kind":"dataset","url":"https://huggingface.co/datasets/xlangai/DS-1000","label":"Source dataset"},{"kind":"issues","url":"https://huggingface.co/datasets/xlangai/DS-1000/discussions","label":"Dataset discussions"},{"kind":"site","url":"https://ds1000-code-gen.github.io/","label":"Official site"},{"kind":"paper","url":"https://arxiv.org/abs/2211.11501","label":"arXiv:2211.11501"}],"task_format":"A data-science problem with the surrounding code context stated in the prompt. The reference code and the checking context are not part of the prompt.","scoring":"Correctness is decided by executing the candidate inside the problem’s own code context, so the reference code the source ships is not a graded answer; it stays in restricted storage.","metric":"accuracy (the source paper reports the best public system at 43.3%)","size":"1,000 problems imported from the default / test selection (the full split).","answer_access":"Columns outside the prompt allowlist, including any answer or reference solution, are separated into restricted storage and never rendered.","license":"cc-by-sa-4.0 (declared on the Hugging Face dataset card)","history":"","construction":"DS-1000 contains 1,000 data-science programming problems derived from real Stack Overflow questions across seven libraries, including NumPy, pandas and Matplotlib. The construction process includes perturbing the source problems so that reproducing an original posted answer is not sufficient for the modified task. Candidate code is evaluated by execution within the problem's checking context, rather than by textual comparison with reference code. The catalog imports 200 problems from the full test selection.","limitations":"The benchmark measures bounded programming tasks in a specific set of data-science libraries, rather than an entire analysis workflow or the scientific validity of an investigation. Perturbing public questions reduces the usefulness of verbatim recall but does not establish that the source material or benchmark itself was absent from training. Execution results also depend on the library environment, so comparisons require compatible dependencies and evaluation code.","saturation":"This catalogue's metric record cites the source paper's best public system at 43.3% accuracy, which leaves substantial headroom in that evaluation. That is a historical result rather than a measurement of the systems available today, and no leaderboard entry is recorded here. It supports the benchmark's original difficulty without establishing its current distance from saturation.","citation":"@article{lai2022ds1000,\n  title={DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation},\n  author={Yuhang Lai and Chengxi Li and Yiming Wang and Tianyi Zhang and Ruiqi Zhong and Luke Zettlemoyer and Scott Wen-tau Yih and Daniel Fried and Sida Wang and Tao Yu},\n  journal={arXiv preprint arXiv:2211.11501},\n  year={2022}\n}","maintainer":"xlangai (source organisation)","released":"Dataset published on Hugging Face 2024-04","why_hard":"The problems come from people who were actually stuck, and the perturbations mean the remembered StackOverflow answer is usually close but wrong.","related":["bigcodebench","humaneval-plus","scicode"],"scores":[]}},"facts":{"slug":"ds-1000","task_count":1000,"image_count":0,"formats":[{"count":1000,"value":"code"}],"answers":[{"count":1000,"value":"published"}],"configs":[{"count":1000,"value":"default"}],"splits":[{"count":1000,"value":"test"}],"waiting_count":1000,"post_count":0,"discussed_count":0,"releases":[{"acquired":"2026-09-12","revision":"4416080ac5cb80bdf7576aefb8f9a0b4d5426a44","release_id":"b578ca080b3c9f81a578cf6853c9ab623abc309f91b5484ddb14059144d3d325"},{"acquired":"2026-09-12","revision":"4416080ac5cb80bdf7576aefb8f9a0b4d5426a44","release_id":"8a121e02617bd45c33813b2109ec40756eee00c1544d5503043d8654b08677e6"},{"acquired":"2026-09-12","revision":"4416080ac5cb80bdf7576aefb8f9a0b4d5426a44","release_id":"4346a40c861b6a72c21a3c4ab5c0cab9a6ab6451e629284c8be512d74f9b2f57"}],"terms":{"license":null,"checked_at":null,"distribution_terms":null,"publisher_requests":null}}},{"benchmark":{"slug":"arena-hard","formal_name":"Arena-Hard-Auto","category":"Instruction following","effective_mode":"metadata_only","profile":{"resources":[{"kind":"dataset","url":"https://huggingface.co/datasets/lmarena-ai/arena-hard-auto-v0.1","label":"Source dataset"},{"kind":"issues","url":"https://huggingface.co/datasets/lmarena-ai/arena-hard-auto-v0.1/discussions","label":"Dataset discussions"},{"kind":"repository","url":"https://github.com/lmarena/arena-hard-auto","label":"Official repository"},{"kind":"paper","url":"https://arxiv.org/abs/2406.11939","label":"arXiv:2406.11939"}],"task_format":"An open-ended user prompt with its topic cluster; answers are compared against a baseline model.","scoring":"There is no reference answer — a judge model compares two responses, so there is nothing to store as an answer.","metric":"","size":"500 prompts in the source; none imported.","answer_access":"No problem text is imported, so there is no answer storage for this entry. The source distributes the answers under its own terms.","license":"apache-2.0 (declared on the Hugging Face dataset card)","history":"The LMArena team developed Arena-Hard-Auto to evaluate responses to difficult prompts drawn from real user conversations using automated preference judgements. The source distribution recorded here is the v0.1 dataset, published under lmarena-ai on Hugging Face in September 2024. Its purpose is to make preference-based comparison more repeatable without collecting fresh human votes for every pair of models.","construction":"Arena-Hard-Auto selects 500 prompts from a pool of 200,000 real LMArena user conversations. The source associates prompts with topic clusters and evaluates model responses against a baseline response. There is no single reference answer: a judge model compares the two responses, so the evaluation depends on the chosen judge and comparison protocol.","limitations":"A model judge measures a proxy for human preference, rather than independently verifying every factual or technical claim in an answer. Response style and length can affect preference judgements, and a change in the judge or baseline changes the comparison being made. The selected difficult prompts also do not represent the frequency of tasks in everyday use. Scores from different Arena-Hard versions or judging configurations therefore need separate interpretation.","saturation":"","citation":"@article{li2024arenahard,\n  title={From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline},\n  author={Tianle Li and Wei-Lin Chiang and Evan Frick and Lisa Dunlap and Tianhao Wu and Banghua Zhu and Joseph E. Gonzalez and Ion Stoica},\n  journal={arXiv preprint arXiv:2406.11939},\n  year={2024}\n}","maintainer":"lmarena-ai (source organisation)","released":"Dataset published on Hugging Face 2024-09","why_hard":"The prompts were chosen for being the ones real users found hard, and they have no single correct answer, so the score depends on judgement quality as much as on the response.","related":["mt-bench","ifeval"],"scores":[]}},"facts":null},{"benchmark":{"slug":"mt-bench","formal_name":"MT-Bench","category":"Instruction following","effective_mode":"metadata_only","profile":{"resources":[{"kind":"dataset","url":"https://huggingface.co/datasets/HuggingFaceH4/mt_bench_prompts","label":"Source dataset"},{"kind":"issues","url":"https://huggingface.co/datasets/HuggingFaceH4/mt_bench_prompts/discussions","label":"Dataset discussions"},{"kind":"repository","url":"https://github.com/lm-sys/FastChat/tree/main/fastchat/llm_judge","label":"Official evaluation code"},{"kind":"paper","url":"https://arxiv.org/abs/2306.05685","label":"arXiv:2306.05685"}],"task_format":"Two successive user turns in one category, the second depending on the answer to the first.","scoring":"Scored by a judge model on a 1–10 scale; there is no reference answer to store for most categories.","metric":"average judge score over the eight categories, reported per turn","size":"80 questions in the source; none imported.","answer_access":"No problem text is imported, so there is no answer storage for this entry. The source distributes the answers under its own terms.","license":"apache-2.0 (declared on the Hugging Face dataset card)","history":"Researchers in the LMSYS group introduced MT-Bench in 2023 as part of their study of language models as judges of assistant responses. They designed it to examine instruction following and conversation across two turns, including dependencies that a single-turn test cannot capture. The HuggingFaceH4 prompt distribution was published in July 2023.","construction":"MT-Bench contains 80 two-turn questions spanning eight categories. The second user turn builds on the first exchange, allowing the evaluation to test whether an assistant follows the conversation as well as the immediate instruction. A judge model assigns scores on a 1-10 scale, with results reported by turn and averaged across categories; most categories do not have a single reference answer.","limitations":"Model-based judging can reward verbosity and exhibit position or self-preference biases, issues examined in the original study. Scores depend on the judge model and judging prompt, and are not interchangeable with objectively verified task accuracy. Eighty short conversations offer limited coverage of extended interaction, while the public prompts can become targets for training or benchmark-specific optimisation.","saturation":"","citation":"@article{zheng2023mtbench,\n  title={Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena},\n  author={Lianmin Zheng and Wei-Lin Chiang and Ying Sheng and Siyuan Zhuang and Zhanghao Wu and Yonghao Zhuang and Zi Lin and Zhuohan Li and Dacheng Li and Eric P. Xing and Hao Zhang and Joseph E. Gonzalez and Ion Stoica},\n  journal={arXiv preprint arXiv:2306.05685},\n  year={2023}\n}","maintainer":"HuggingFaceH4 (source organisation)","released":"Dataset published on Hugging Face 2023-07","why_hard":"The second turn depends on the first, so an answer that was locally fine but unhelpful for what follows is penalised where a single-turn benchmark would not notice.","related":["arena-hard","ifeval"],"scores":[]}},"facts":null},{"benchmark":{"slug":"simpleqa","formal_name":"SimpleQA","category":"Knowledge / Search","effective_mode":"full","profile":{"resources":[{"kind":"dataset","url":"https://huggingface.co/datasets/basicv8vc/SimpleQA","label":"Source dataset"},{"kind":"issues","url":"https://huggingface.co/datasets/basicv8vc/SimpleQA/discussions","label":"Dataset discussions"},{"kind":"repository","url":"https://github.com/openai/simple-evals","label":"Official evaluation code"},{"kind":"paper","url":"https://arxiv.org/abs/2411.04368","label":"arXiv:2411.04368"}],"task_format":"One short question. The reference answer and the URLs that document it are not part of the prompt.","scoring":"The source ships a reference answer with every question; this catalogue keeps it in restricted storage and never renders it.","metric":"correct, incorrect and not-attempted rates, and the F-score that combines them","size":"4,326 problems imported from the default / test selection (the full split).","answer_access":"Columns outside the prompt allowlist, including any answer or reference solution, are separated into restricted storage and never rendered.","license":"mit (declared on the Hugging Face dataset card)","history":"OpenAI introduced SimpleQA in 2024 to measure factual accuracy on short questions with narrowly specified answers. It was designed to make factual grading more tractable than evaluation of long responses containing many separate claims. Its evaluation distinguishes correct answers, incorrect answers, and cases where a model does not attempt an answer.","construction":"SimpleQA was assembled from human-written fact-seeking questions with reference answers and supporting sources. The questions were designed to have a single verifiable answer rather than require an open-ended explanation. The source contains 4,326 questions; this catalogue imports 200 test records and keeps both reference answers and documenting URLs outside the prompt.","limitations":"Short factual questions do not measure whether a long response remains accurate across many claims, or whether a system can synthesize a useful explanation. The intended answerability of the questions does not eliminate possible ambiguity or changes in the underlying facts. Access to search or other retrieval tools changes the capability being evaluated, so tool-assisted and closed-book results need separate interpretation. Correct, incorrect, and not-attempted rates also carry information that a single aggregate score can conceal.","saturation":"","citation":"@article{wei2024simpleqa,\n  title={Measuring short-form factuality in large language models},\n  author={Jason Wei and Nguyen Karina and Hyung Won Chung and Yunxin Joy Jiao and Spencer Papay and Amelia Glaese and John Schulman and William Fedus},\n  journal={arXiv preprint arXiv:2411.04368},\n  year={2024}\n}","maintainer":"basicv8vc (source organisation)","released":"Dataset published on Hugging Face 2024-11","why_hard":"The questions are short but the facts are obscure, and a model that guesses scores worse than one that abstains. It measures calibration as much as knowledge.","related":["browsecomp","assistantbench","browsecomp-plus"],"scores":[]}},"facts":{"slug":"simpleqa","task_count":4326,"image_count":0,"formats":[{"count":4326,"value":"text"}],"answers":[{"count":4325,"value":"published"},{"count":1,"value":"unavailable"}],"configs":[{"count":4326,"value":"default"}],"splits":[{"count":4326,"value":"test"}],"waiting_count":4326,"post_count":0,"discussed_count":0,"releases":[{"acquired":"2026-09-12","revision":"e319282ab125c3dbd0c7fd00be2e4dd54e7e8f94","release_id":"2f00e67ed0ebbbf4f0c7ad7c2e405837b29933af02cce1a6e09fea3bd94e2a92"},{"acquired":"2026-09-12","revision":"e319282ab125c3dbd0c7fd00be2e4dd54e7e8f94","release_id":"2f3ac1c4c2668336d032f590623e77e1e610b59469f0ddfc9f5c6019d4df1514"}],"terms":{"license":null,"checked_at":null,"distribution_terms":null,"publisher_requests":null}}},{"benchmark":{"slug":"aime-2024","formal_name":"AIME 2024","category":"Mathematics","effective_mode":"full","profile":{"resources":[{"kind":"dataset","url":"https://huggingface.co/datasets/Maxwell-Jia/AIME_2024","label":"Source dataset"},{"kind":"issues","url":"https://huggingface.co/datasets/Maxwell-Jia/AIME_2024/discussions","label":"Dataset discussions"}],"task_format":"One competition problem statement. The official solution and the integer answer are not part of the prompt.","scoring":"The source ships the worked solution and the integer answer; this catalogue keeps both in restricted storage and never renders them.","metric":"","size":"All 30 problems imported from the default / train selection, with a source-row limit of 200.","answer_access":"Columns outside the prompt allowlist, including any answer or reference solution, are separated into restricted storage and never rendered.","license":"mit (declared on the Hugging Face dataset card)","history":"The 2024 American Invitational Mathematics Examination was a mathematics competition administered by the Mathematical Association of America, rather than a test originally designed for language models. Its short numerical answers and demanding solutions made it useful for evaluating mathematical reasoning. Maxwell-Jia published the dataset version recorded here on Hugging Face in December 2024.","construction":"The dataset collects all 30 problems from the 2024 AIME I and AIME II examinations. These are human-authored competition problems, each requiring an integer answer from 0 to 999 rather than a choice among supplied options. The source includes worked solutions and final answers, which are excluded from the evaluation prompt.","limitations":"Thirty problems provide a small sample of mathematical ability: one additional correct answer changes single-run accuracy by about 3.3 percentage points. Final-answer grading gives no partial credit and does not establish that the accompanying reasoning is valid. The public availability of the problems and solutions creates a potential training-contamination risk, while sampling, output limits and repeated attempts can change reported performance.","saturation":"The vendor report recorded here gives DeepSeek V3 a pass@1 score of 39.2 percent, using an 8K output limit and repeated runs with varying temperatures for small benchmarks. That result leaves substantial headroom in the reported setup, but it does not establish the present best performance. Fractional scores from repeated sampling should not be interpreted as the number of problems solved in a single 30-question attempt.","citation":"","maintainer":"Maxwell-Jia (source organisation)","released":"Dataset published on Hugging Face 2024-12","why_hard":"Each problem takes a multi-step construction that has to be carried out exactly. A single arithmetic slip produces a wrong integer, and thirty problems make each one worth three points of the score.","related":["aime-2025","amc","math"],"scores":[{"model":"DeepSeek V3","organization":"DeepSeek","score":39.2,"metric":"Pass@1","setup":"Chat model; 8K output limit; multiple runs with varying temperatures for benchmarks under 1000 samples.","reported_by":"vendor","source_url":"https://raw.githubusercontent.com/deepseek-ai/DeepSeek-V3/main/README.md","source_label":"Chat Model, Standard Benchmarks, AIME 2024 (Pass@1), final column","reported_at":"","checked_at":"2026-09-16"}]}},"facts":{"slug":"aime-2024","task_count":30,"image_count":0,"formats":[{"count":30,"value":"math"}],"answers":[{"count":30,"value":"published"}],"configs":[{"count":30,"value":"default"}],"splits":[{"count":30,"value":"train"}],"waiting_count":30,"post_count":0,"discussed_count":0,"releases":[{"acquired":"2026-09-12","revision":"8d88b2876a82a080e2f172cc9b25d0d9d2cb4792","release_id":"31c7e41d93d9134bbf3b4378580856ba2ef284a741108e608805b5300597ebe5"},{"acquired":"2026-09-12","revision":"8d88b2876a82a080e2f172cc9b25d0d9d2cb4792","release_id":"a0ba82c12c7089cc7863c9ba934fb9ba6ed7089967a3f989847e530309286069"},{"acquired":"2026-09-12","revision":"8d88b2876a82a080e2f172cc9b25d0d9d2cb4792","release_id":"9986a9443c459cd15c191fef03212ab46c55b34cc07e029fbe8defef12117f56"}],"terms":{"license":null,"checked_at":null,"distribution_terms":null,"publisher_requests":null}}},{"benchmark":{"slug":"aime-2025","formal_name":"AIME 2025 I","category":"Mathematics","effective_mode":"full","profile":{"resources":[{"kind":"dataset","url":"https://huggingface.co/datasets/opencompass/AIME2025","label":"Source dataset"},{"kind":"issues","url":"https://huggingface.co/datasets/opencompass/AIME2025/discussions","label":"Dataset discussions"}],"task_format":"One competition problem statement. The integer answer is not part of the prompt.","scoring":"The source ships the integer answer with every problem; this catalogue keeps it in restricted storage and never renders it.","metric":"","size":"All 15 problems imported from the AIME2025-I / test selection, with a source-row limit of 200.","answer_access":"Columns outside the prompt allowlist, including any answer or reference solution, are separated into restricted storage and never rendered.","license":"mit (declared on the Hugging Face dataset card)","history":"The Mathematical Association of America produced the 2025 American Invitational Mathematics Examination as part of its mathematics competition program. The problems were intended to distinguish strong competition students through demanding, short-answer mathematics. OpenCompass distributed a machine-readable dataset on Hugging Face in February 2025, making the examination available for evaluating mathematical reasoning in language models.","construction":"This entry covers the 15 problems from AIME I 2025, rather than a combined collection of both AIME examinations that year. The problems are human-authored competition questions with integer answers, and the source dataset supplies an answer for each one. Evaluation can compare a model's extracted final answer with that integer, without requiring its reasoning to match a reference derivation.","limitations":"Fifteen questions provide a small sample: one additional correct answer changes the percentage score substantially. Exact final-answer grading does not establish that a derivation is valid, and competition mathematics does not cover the full range of mathematical practice. The 2025 date can reduce exposure for models trained before the examination, but it does not establish freedom from contamination for models trained afterward. Reports also need to distinguish AIME I from combined AIME I and II evaluations and specify sampling and reasoning budgets.","saturation":"","citation":"","maintainer":"opencompass (source organisation)","released":"Dataset published on Hugging Face 2025-02","why_hard":"Same demand as any AIME paper — an exact integer after a long construction — with the added property that the problems postdate most training corpora.","related":["aime-2024","amc","math"],"scores":[]}},"facts":{"slug":"aime-2025","task_count":15,"image_count":0,"formats":[{"count":15,"value":"math"}],"answers":[{"count":15,"value":"published"}],"configs":[{"count":15,"value":"AIME2025-I"}],"splits":[{"count":15,"value":"test"}],"waiting_count":15,"post_count":0,"discussed_count":0,"releases":[{"acquired":"2026-09-12","revision":"a6ad95f611d72cf628a80b58bd0432ef6638f958","release_id":"a9b002df8349a46afbb78e4eed5a8a26e62c723f8b53703ab021b5187b17bfcf"},{"acquired":"2026-09-12","revision":"a6ad95f611d72cf628a80b58bd0432ef6638f958","release_id":"670d19decaae4fdc9f98aa16c4498fd504fc77a557602df7b602ce423cf7a701"},{"acquired":"2026-09-12","revision":"a6ad95f611d72cf628a80b58bd0432ef6638f958","release_id":"487ae737f06618392b081eb31728b262844864ba57d20714626a8b4fe85fa721"}],"terms":{"license":null,"checked_at":null,"distribution_terms":null,"publisher_requests":null}}},{"benchmark":{"slug":"amc","formal_name":"AMC (AIMO validation set)","category":"Mathematics","effective_mode":"full","profile":{"resources":[{"kind":"dataset","url":"https://huggingface.co/datasets/AI-MO/aimo-validation-amc","label":"Source dataset"},{"kind":"issues","url":"https://huggingface.co/datasets/AI-MO/aimo-validation-amc/discussions","label":"Dataset discussions"}],"task_format":"One competition problem statement. The numeric answer is not part of the prompt, and neither is the Art of Problem Solving page, which is recorded as the problem’s upstream identifier because that page also carries a solution.","scoring":"The source ships the numeric answer with every problem; this catalogue keeps it in restricted storage and never renders it.","metric":"","size":"All 83 problems imported from the default / train selection, with a source-row limit of 200.","answer_access":"Columns outside the prompt allowlist, including any answer or reference solution, are separated into restricted storage and never rendered.","license":"apache-2.0 (declared on the Hugging Face dataset card)","history":"Project Numina assembled this AMC validation set for the AIMO competition, publishing it under AI-MO on Hugging Face in July 2024. It uses AMC 12 competition problems to assess mathematical problem solving at a level below AIME, providing a less demanding complement to harder competition evaluations.","construction":"The set contains 83 existing AMC 12 problems, rather than questions newly written for evaluating language models. Each record pairs a problem statement with a numeric answer and identifies an upstream Art of Problem Solving page. The catalogue includes all 83 problems, while excluding answers and solution-bearing upstream pages from the prompt.","limitations":"This is a small selection of competition mathematics, so a handful of outcomes can substantially change an aggregate score. Its existing problems and publicly available solution pages create a potential route for training-data contamination, although their availability alone does not establish exposure by any particular model. Numeric-answer evaluation also leaves the validity of the intermediate reasoning unmeasured.","saturation":"","citation":"","maintainer":"AI-MO / Project Numina (source organisation)","released":"Dataset published on Hugging Face 2024-07","why_hard":"AMC problems reward a single clean idea rather than a long construction, so failures here point at missing technique rather than at lost arithmetic.","related":["aime-2024","aime-2025","math"],"scores":[]}},"facts":{"slug":"amc","task_count":83,"image_count":0,"formats":[{"count":83,"value":"math"}],"answers":[{"count":83,"value":"published"}],"configs":[{"count":83,"value":"default"}],"splits":[{"count":83,"value":"train"}],"waiting_count":83,"post_count":0,"discussed_count":0,"releases":[{"acquired":"2026-09-12","revision":"69d78a4a2c840e82d69af6bc742bda09005f6316","release_id":"f88229dded3567a66c266287d82cb6031e432bf0207625d374db5e27352a2372"},{"acquired":"2026-09-12","revision":"69d78a4a2c840e82d69af6bc742bda09005f6316","release_id":"4078a47d8a628c736f70ece2ed322a02da7cf5d7abc791a8327b517e0245e3d5"},{"acquired":"2026-09-12","revision":"69d78a4a2c840e82d69af6bc742bda09005f6316","release_id":"920cd2975153765a1180d306dfae426d76206e85aec745b354f2b3a3fa6f604d"}],"terms":{"license":null,"checked_at":null,"distribution_terms":null,"publisher_requests":null}}},{"benchmark":{"slug":"frontiermath","formal_name":"FrontierMath","category":"Mathematics","effective_mode":"metadata_only","profile":{"resources":[{"kind":"site","url":"https://epoch.ai/frontiermath","label":"Official site"}],"task_format":"Advanced mathematical problems written by experts.","scoring":"","metric":"","size":"","answer_access":"This catalog excludes private evaluation problems and provides official introduction and public-example links only.","license":"","history":"Epoch AI introduced FrontierMath in 2024 to evaluate advanced mathematical reasoning with problems written by expert mathematicians. The project addressed the need for difficult mathematical evaluations beyond familiar competition-style questions. Its private evaluation problems are distinct from the public examples used to illustrate the benchmark.","construction":"FrontierMath uses expert-written mathematical problems rather than simply collecting an existing public examination. The evaluation includes private problems, while public examples provide a limited view of the task format and difficulty. This catalogue links to the official introduction and examples and does not reproduce the private evaluation set.","limitations":"Success on advanced mathematical problems does not by itself measure the broader process of mathematical research, such as choosing worthwhile questions or developing a reusable theory. The private evaluation set also limits independent inspection of its coverage and individual outcomes. Public examples cannot establish how representative a reported score is of other mathematical domains.","saturation":"","citation":"","maintainer":"","released":"","why_hard":"Requires advanced mathematical reasoning on expert-written problems.","related":["omni-math","olympiadbench"],"scores":[]}},"facts":null},{"benchmark":{"slug":"gsm8k","formal_name":"GSM8K","category":"Mathematics","effective_mode":"full","profile":{"resources":[{"kind":"dataset","url":"https://huggingface.co/datasets/openai/gsm8k","label":"Source dataset"},{"kind":"issues","url":"https://huggingface.co/datasets/openai/gsm8k/discussions","label":"Dataset discussions"},{"kind":"paper","url":"https://arxiv.org/abs/2110.14168","label":"arXiv:2110.14168"}],"task_format":"A short word problem in the question field. The worked solution and final answer are kept out of the prompt.","scoring":"The source ships an answer with every problem; this catalogue keeps it in restricted storage and never renders it.","metric":"","size":"1,319 problems imported from the main / test selection (the full test split).","answer_access":"Columns outside the prompt allowlist, including any answer or reference solution, are separated into restricted storage and never rendered.","license":"mit (declared on the source)","history":"OpenAI introduced GSM8K in 2021 as a benchmark of multistep mathematical reasoning expressed in ordinary language. The work examined whether models could produce and assess solutions to grade-school word problems, where individual calculations are elementary but must be combined correctly. The April 2022 Hugging Face publication recorded for this source is the date of the hosted source, not the benchmark's original introduction.","construction":"GSM8K consists of 8,792 human-written grade-school mathematics word problems with worked solutions. The standard split contains 7,473 training problems and 1,319 test problems. Problems require several dependent arithmetic steps, and evaluation typically extracts the final numerical answer from the generated solution. This catalogue imports 200 problems from the main test split, rather than the complete test set.","limitations":"GSM8K covers a restricted range of short arithmetic word problems and does not measure advanced mathematics, proof construction or visual reasoning. Final-answer exact match does not verify each step of a solution, and answer extraction can affect the measured score. Because both the problems and worked solutions are publicly distributed, benchmark accuracy alone cannot distinguish general reasoning from prior exposure. Results also depend on whether models use tools, multiple samples or selection among candidate solutions.","saturation":"The vendor report recorded here records 89.3 exact match for the DeepSeek-V3 base model in an eight-shot setup. That result demonstrates strong performance while leaving errors in the reported configuration; it does not establish the ceiling reached by other systems. The remaining gap should be interpreted alongside the evaluation setup and the benchmark's narrow arithmetic coverage, rather than as a general measure of mathematical capability.","citation":"@article{cobbe2021gsm8k,\n  title={Training Verifiers to Solve Math Word Problems},\n  author={Cobbe, Karl and Kosaraju, Vineet and Bavarian, Mohammad and Chen, Mark and Jun, Heewoo and Kaiser, Lukasz and Plappert, Matthias and Tworek, Jerry and Hilton, Jacob and Nakano, Reiichiro and Hesse, Christopher and Schulman, John},\n  journal={arXiv preprint arXiv:2110.14168},\n  year={2021}\n}","maintainer":"openai (source organisation)","released":"Dataset published on Hugging Face 2022-04","why_hard":"Each problem is elementary in isolation; the difficulty is carrying several dependent steps without arithmetic slips.","related":["frontiermath","olympiadbench","omni-math"],"scores":[{"model":"DeepSeek-V3","organization":"DeepSeek","score":89.3,"metric":"EM","setup":"Base model; 8-shot.","reported_by":"vendor","source_url":"https://raw.githubusercontent.com/deepseek-ai/DeepSeek-V3/main/README.md","source_label":"Base Model, Standard Benchmarks, GSM8K (EM), final column","reported_at":"","checked_at":"2026-09-16"}]}},"facts":{"slug":"gsm8k","task_count":1319,"image_count":0,"formats":[{"count":1319,"value":"text"}],"answers":[{"count":1319,"value":"published"}],"configs":[{"count":1319,"value":"main"}],"splits":[{"count":1319,"value":"test"}],"waiting_count":1319,"post_count":0,"discussed_count":0,"releases":[{"acquired":"2026-09-11","revision":"740312add88f781978c0658806c59bc2815b9866","release_id":"a81f6d44d3186ea648c7bfd5e4ca6801d8f5fe1b7beb56990437748eefef1867"},{"acquired":"2026-09-11","revision":"740312add88f781978c0658806c59bc2815b9866","release_id":"c754bdfb5f7c0bee6e7f9f7b5f547fdd1fc30d07bc6c856b647be678938bd8b1"},{"acquired":"2026-09-11","revision":"740312add88f781978c0658806c59bc2815b9866","release_id":"a8f963ca5e1e5aa8f1dc8cbc0934d963f6657045f850c252d43ae2143840e415"}],"terms":{"license":null,"checked_at":null,"distribution_terms":null,"publisher_requests":null}}},{"benchmark":{"slug":"math","formal_name":"MATH","category":"Mathematics","effective_mode":"full","profile":{"resources":[{"kind":"dataset","url":"https://huggingface.co/datasets/EleutherAI/hendrycks_math","label":"Source dataset"},{"kind":"issues","url":"https://huggingface.co/datasets/EleutherAI/hendrycks_math/discussions","label":"Dataset discussions"},{"kind":"repository","url":"https://github.com/hendrycks/math","label":"Official repository"},{"kind":"paper","url":"https://arxiv.org/abs/2103.03874","label":"arXiv:2103.03874"}],"task_format":"A problem statement in LaTeX, with its subject and difficulty level. The worked solution is not part of the prompt.","scoring":"The source ships a step-by-step solution with every problem; this catalogue keeps it in restricted storage and never renders it.","metric":"accuracy on the final boxed answer","size":"5,000 problems imported from the test split across all seven subjects.","answer_access":"Columns outside the prompt allowlist, including any answer or reference solution, are separated into restricted storage and never rendered.","license":"mit (declared on the Hugging Face dataset card)","history":"Dan Hendrycks and collaborators introduced MATH in 2021 to test mathematical problem solving beyond routine word problems. They assembled competition mathematics with worked solutions so that evaluation could address harder reasoning and training could use intermediate steps. EleutherAI maintains the Hugging Face distribution recorded here, whose 2023 publication date is later than the original benchmark.","construction":"MATH collects 12,500 high-school competition problems with worked solutions, organised into seven subjects and five difficulty levels. The material comes from existing competition problem collections rather than newly generated model outputs. The catalogue contains a 200-problem selection from the algebra test split, whose source contains 1,187 problems; that selection is not the full benchmark.","limitations":"Final-answer accuracy does not establish that the reasoning leading to an answer is valid. Answer extraction and equivalence checking can affect whether mathematically equivalent responses receive credit. Public problems and worked solutions create a risk of training exposure. Results on an algebra selection or another reduced subset should be distinguished from results on the full test set.","saturation":"The recorded vendor report gives DeepSeek-V3 base 61.6 percent exact match with four examples in the prompt. This leaves considerable headroom for that setup, but one base-model report is insufficient to establish the ceiling for stronger reasoning systems. Comparisons need to identify the evaluated subset and whether the model received tools or multiple attempts.","citation":"@article{hendrycks2021math,\n  title={Measuring Mathematical Problem Solving With the MATH Dataset},\n  author={Dan Hendrycks and Collin Burns and Saurav Kadavath and Akul Arora and Steven Basart and Eric Tang and Dawn Song and Jacob Steinhardt},\n  journal={arXiv preprint arXiv:2103.03874},\n  year={2021}\n}","maintainer":"EleutherAI (source organisation)","released":"Dataset published on Hugging Face 2023-09","why_hard":"The problems are written for students who have the technique and need to choose it. Difficulty is stated per problem, so level 5 items are hard by construction rather than by accident.","related":["aime-2024","amc","gsm8k"],"scores":[{"model":"DeepSeek-V3","organization":"DeepSeek","score":61.6,"metric":"EM","setup":"Base model; 4-shot.","reported_by":"vendor","source_url":"https://raw.githubusercontent.com/deepseek-ai/DeepSeek-V3/main/README.md","source_label":"Base Model, Standard Benchmarks, MATH (EM), final column","reported_at":"","checked_at":"2026-09-16"}]}},"facts":{"slug":"math","task_count":5000,"image_count":0,"formats":[{"count":5000,"value":"math"}],"answers":[{"count":5000,"value":"published"}],"configs":[{"count":1187,"value":"algebra"},{"count":474,"value":"counting_and_probability"},{"count":479,"value":"geometry"},{"count":903,"value":"intermediate_algebra"},{"count":540,"value":"number_theory"},{"count":871,"value":"prealgebra"},{"count":546,"value":"precalculus"}],"splits":[{"count":5000,"value":"test"}],"waiting_count":5000,"post_count":0,"discussed_count":0,"releases":[{"acquired":"2026-09-23","revision":"21a5633873b6a120296cce3e2df9d5550074f4a3","release_id":"36bf50fafb0292922e65b155d5d990dcc7e2ed6d923450861cc36d0a10c35aca"},{"acquired":"2026-09-12","revision":"21a5633873b6a120296cce3e2df9d5550074f4a3","release_id":"ae4d22a7a5ac8da1d3f0fd92f3074c0e530a0358999652dd0fa5cad21cb6eaa9"},{"acquired":"2026-09-12","revision":"21a5633873b6a120296cce3e2df9d5550074f4a3","release_id":"7febcb3508c05bc1cb6bbbeb477e2fc29e1de155ffc5b642be540dfca9ac8bce"}],"terms":{"license":null,"checked_at":null,"distribution_terms":null,"publisher_requests":null}}},{"benchmark":{"slug":"omni-math","formal_name":"Omni-MATH","category":"Mathematics","effective_mode":"full","profile":{"resources":[{"kind":"dataset","url":"https://huggingface.co/datasets/KbsdJames/Omni-MATH","label":"Source dataset"},{"kind":"issues","url":"https://huggingface.co/datasets/KbsdJames/Omni-MATH/discussions","label":"Dataset discussions"},{"kind":"paper","url":"https://arxiv.org/abs/2410.07985","label":"arXiv:2410.07985"}],"task_format":"Olympiad-level mathematical problem statements; the generic import accepts the scalar problem field.","scoring":"The source ships an answer with every problem; this catalogue keeps it in restricted storage and never renders it. The authors publish Omni-Judge, an open evaluator for this dataset, at https://huggingface.co/KbsdJames/Omni-Judge.","metric":"","size":"4,406 unique problems imported from the test selection of 4,428 rows; 22 rows repeat an earlier problem and are recorded as variants.","answer_access":"Fields outside the generic prompt allowlist, including answers and solutions when supplied, remain in restricted storage; the converter alone does not establish whether usable answers are available.","license":"apache-2.0 (declared on the Hugging Face dataset card)","history":"The Omni-MATH project introduced its Olympiad-level mathematics benchmark in 2024, with this catalogue recording a September Hugging Face publication under KbsdJames. It targeted mathematical reasoning on difficult competition problems. Domain and difficulty annotations support analysis beyond a single aggregate accuracy figure.","construction":"The official dataset contains 4,428 Olympiad-level mathematical problems accompanied by domain and difficulty information. The distribution supplies answers, and the authors also publish an evaluator called Omni-Judge. The catalogue uses a capped selection of 200 task records rather than the entire collection.","limitations":"Olympiad-level mathematics is a specialised domain and does not directly measure mathematical modelling, empirical work or formal proof verification. A correct final answer alone is not evidence that every step of a solution is valid. Reports need to identify the evaluator and any tool or sampling budget, since those choices affect what a success rate represents.","saturation":"","citation":"@misc{gao2024omnimathuniversalolympiadlevel,\n      title={Omni-MATH: A Universal Olympiad Level Mathematic Benchmark For Large Language Models}, \n      author={Bofei Gao and Feifan Song and Zhe Yang and Zefan Cai and Yibo Miao and Qingxiu Dong and Lei Li and Chenghao Ma and Liang Chen and Runxin Xu and Zhengyang Tang and Benyou Wang and Daoguang Zan and Shanghaoran Quan and Ge Zhang and Lei Sha and Yichang Zhang and Xuancheng Ren and Tianyu Liu and Baobao Chang},\n      year={2024},\n      eprint={2410.07985},\n      archivePrefix={arXiv},\n      primaryClass={cs.CL},\n      url={https://arxiv.org/abs/2410.07985}, \n}","maintainer":"KbsdJames (Hugging Face organisation)","released":"Dataset published on Hugging Face 2024-09","why_hard":"Requires mathematical reasoning on Olympiad-level problems.","related":["olympiadbench","frontiermath"],"scores":[]}},"facts":{"slug":"omni-math","task_count":4406,"image_count":0,"formats":[{"count":4206,"value":"math"},{"count":200,"value":"text"}],"answers":[{"count":4406,"value":"published"}],"configs":[{"count":4406,"value":""}],"splits":[{"count":4406,"value":"test"}],"waiting_count":4406,"post_count":0,"discussed_count":0,"releases":[{"acquired":"2026-09-09","revision":"40ba231d8f16e29ecd40e6407e2c8640145a8f62","release_id":"2ee237636774effbb02acf5440d413c38a75cfe1ea9df806236218ddd659016c"},{"acquired":"2026-09-09","revision":"40ba231d8f16e29ecd40e6407e2c8640145a8f62","release_id":"27a71f2fd34aac5d8146f95e537299fadaf6a5879f00f9f7fad36bc05aed2e86"},{"acquired":"2026-09-09","revision":"40ba231d8f16e29ecd40e6407e2c8640145a8f62","release_id":"9933a682d064a271ff7fccf9322ce10daa5a9ef73fef6043d3997553c896f404"},{"acquired":"2026-09-09","revision":"40ba231d8f16e29ecd40e6407e2c8640145a8f62","release_id":"e02b298818a82c470c8b4982a57f4c71b7653525efdeb6ee4b5a4db34ccb12e4"},{"acquired":"2026-09-08","revision":"40ba231d8f16e29ecd40e6407e2c8640145a8f62","release_id":"d2fdbb3503e84f6a424497ce7692fed76aaea14b741085c4f5b7f44bb42d0565"}],"terms":{"license":null,"checked_at":null,"distribution_terms":null,"publisher_requests":null}}},{"benchmark":{"slug":"mgsm","formal_name":"MGSM","category":"Mathematics / Multilingual","effective_mode":"full","profile":{"resources":[{"kind":"dataset","url":"https://github.com/google-research/url-nlp/tree/main/mgsm","label":"Original TSV distribution"},{"kind":"repository","url":"https://github.com/google-research/url-nlp","label":"Official repository"},{"kind":"paper","url":"https://arxiv.org/abs/2210.03057","label":"arXiv:2210.03057"},{"kind":"paper","url":"https://arxiv.org/abs/2110.14168","label":"arXiv:2110.14168 (GSM8K, the dataset this translates)"}],"task_format":"One grade-school word problem in the selected language. The answer, the numeric answer and the worked equation are not part of the prompt.","scoring":"The source ships the answer, the numeric answer and the equation; this catalogue keeps all three in restricted storage and never renders them.","metric":"","size":"2,750 problems imported from the original TSVs: the same 250 grade-school problems in each of the eleven languages, kept as per-language pages.","answer_access":"Columns outside the prompt allowlist, including any answer or reference solution, are separated into restricted storage and never rendered.","license":"cc-by-4.0 (declared in the original repository mgsm/LICENSE)","history":"Freda Shi and collaborators at Google Research introduced MGSM in 2022 to investigate whether chain-of-thought reasoning could work across languages. They adapted grade-school mathematics problems from GSM8K so that the same underlying reasoning task could be compared across language settings. The Hugging Face source listed in the catalogue was published in May 2023, after the original research.","construction":"MGSM uses human translations of 250 GSM8K problems and covers eleven languages, including English. The parallel problems preserve the underlying arithmetic while changing the language in which the task is presented. The source supplies answers, numeric answers, and worked equations; this catalogue imports 200 records from the English test selection and separates those reference fields from the prompt.","limitations":"The arithmetic is elementary, so MGSM does not measure advanced mathematical reasoning or broad multilingual competence. Parallel translations support controlled comparisons, but wording and translation choices can affect difficulty as well as model language ability. The English-only catalogue selection cannot by itself measure a multilingual performance gap. Shared problems from the public GSM8K source also create a potential path for training-data overlap.","saturation":"The catalogue records a vendor-reported exact-match score of 79.8 for the DeepSeek-V3 base model with eight-shot prompting. That result remains below the ceiling under the reported setup, but a single recorded score cannot establish the best available performance. Language-level breakdowns and matched prompting are needed to assess whether remaining errors concentrate in particular languages.","citation":"@article{shi2022mgsm,\n  title={Language Models are Multilingual Chain-of-Thought Reasoners},\n  author={Freda Shi and Mirac Suzgun and Markus Freitag and Xuezhi Wang and Suraj Srivats and Soroush Vosoughi and Hyung Won Chung and Yi Tay and Sebastian Ruder and Denny Zhou and Dipanjan Das and Jason Wei},\n  journal={arXiv preprint arXiv:2210.03057},\n  year={2022}\n}","maintainer":"juletxara (source organisation)","released":"Dataset published on Hugging Face 2023-05","why_hard":"The arithmetic is elementary, so anything lost against the English score is lost in the language, not the mathematics — including in languages with little presence in training data.","related":["gsm8k","math","olympiadbench"],"scores":[{"model":"DeepSeek-V3","organization":"DeepSeek","score":79.8,"metric":"EM","setup":"Base model; 8-shot.","reported_by":"vendor","source_url":"https://raw.githubusercontent.com/deepseek-ai/DeepSeek-V3/main/README.md","source_label":"Base Model, Standard Benchmarks, MGSM (EM), final column","reported_at":"","checked_at":"2026-09-16"}]}},"facts":{"slug":"mgsm","task_count":2750,"image_count":0,"formats":[{"count":2750,"value":"math"}],"answers":[{"count":2750,"value":"published"}],"configs":[{"count":250,"value":"bn"},{"count":250,"value":"de"},{"count":250,"value":"en"},{"count":250,"value":"es"},{"count":250,"value":"fr"},{"count":250,"value":"ja"},{"count":250,"value":"ru"},{"count":250,"value":"sw"},{"count":250,"value":"te"},{"count":250,"value":"th"},{"count":250,"value":"zh"}],"splits":[{"count":2750,"value":"test"}],"waiting_count":2750,"post_count":0,"discussed_count":0,"releases":[{"acquired":"2026-09-23","revision":"452a21ad3dae5668c06ceeac21ff073e1e40f9be","release_id":"1017bb4fe10be1e0b20fab5ee93a278fa3101aef67b10d2ad0cfdf2ca8f1020d"},{"acquired":"2026-09-12","revision":"b2f13d426afe3be8d69a7e739b36724db8b66bbc","release_id":"d28706c95f1e03203d880fa0652e8810dd653b407d94425a687b07253fbc8fbb"},{"acquired":"2026-09-12","revision":"b2f13d426afe3be8d69a7e739b36724db8b66bbc","release_id":"a7c8fef5f04f0cea4bc13031dcc7eeb87dce408150daf602bff463dc028962ca"}],"terms":{"license":null,"checked_at":null,"distribution_terms":null,"publisher_requests":null}}},{"benchmark":{"slug":"mathvista","formal_name":"MathVista","category":"Mathematics / Multimodal","effective_mode":"metadata_only","profile":{"resources":[{"kind":"dataset","url":"https://huggingface.co/datasets/AI4Math/MathVista","label":"Source dataset"},{"kind":"issues","url":"https://huggingface.co/datasets/AI4Math/MathVista/discussions","label":"Dataset discussions"},{"kind":"site","url":"https://mathvista.github.io/","label":"Official site"},{"kind":"paper","url":"https://arxiv.org/abs/2310.02255","label":"arXiv:2310.02255"}],"task_format":"A mathematical question with an accompanying image, in either multiple-choice or free-form answer format.","scoring":"The source ships the answer; nothing is imported here.","metric":"accuracy over testmini and the full test set","size":"6,141 problems in the source, 1,000 in testmini; none imported.","answer_access":"No problem text is imported, so there is no answer storage for this entry. The source distributes the answers under its own terms.","license":"cc-by-sa-4.0 (declared on the Hugging Face dataset card)","history":"The MathVista research team introduced the benchmark in 2023 to evaluate mathematical reasoning that depends on visual information. The AI4Math source distribution recorded here was published on Hugging Face in October of that year. Its motivation was to test whether models could connect evidence in charts, figures and geometric diagrams to the calculations or deductions needed for an answer.","construction":"MathVista combines problems from existing visual question-answering resources with newly collected tasks focused on mathematical reasoning. The source contains 6,141 problems, with 1,000 in the public testmini split. Each item pairs an image with a mathematical question, using either multiple-choice or free-form answers, and evaluation measures answer accuracy.","limitations":"A final-answer score combines visual perception, mathematical reasoning and answer extraction, so it does not identify which stage caused an error. The mixture of problem types also means that an aggregate score can hide weaknesses on particular visual formats or mathematical skills. Results on the public 1,000-question testmini split should be distinguished from results on the separate test split, and public examples create a potential exposure risk.","saturation":"","citation":"@article{lu2023mathvista,\n  title={MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts},\n  author={Pan Lu and Hritik Bansal and Tony Xia and Jiacheng Liu and Chunyuan Li and Hannaneh Hajishirzi and Hao Cheng and Kai-Wei Chang and Michel Galley and Jianfeng Gao},\n  journal={arXiv preprint arXiv:2310.02255},\n  year={2023}\n}","maintainer":"AI4Math (source organisation)","released":"Dataset published on Hugging Face 2023-10","why_hard":"The quantities are in the picture. Reading a chart or a geometric figure precisely enough to compute with it is a different skill from solving the equation afterwards.","related":["mmmu","mmmu-pro","math"],"scores":[]}},"facts":null},{"benchmark":{"slug":"olympiadbench","formal_name":"OlympiadBench","category":"Mathematics / Science / Multimodal","effective_mode":"full","profile":{"resources":[{"kind":"dataset","url":"https://huggingface.co/datasets/Hothan/OlympiadBench","label":"Source dataset"},{"kind":"issues","url":"https://huggingface.co/datasets/Hothan/OlympiadBench/discussions","label":"Dataset discussions"},{"kind":"repository","url":"https://github.com/OpenBMB/OlympiadBench","label":"Official repository"},{"kind":"paper","url":"https://arxiv.org/abs/2402.14008","label":"arXiv:2402.14008"}],"task_format":"The selected OE_TO_maths_en_COMP setting contains text-only mathematics questions. The import accepts question, language, subject, answer_type, question_type, unit, and is_multiple_answer.","scoring":"For the 200 imported problems, the source ships an answer with every problem; this catalogue keeps it in restricted storage and never renders it.","metric":"","size":"8,476 English and Chinese problems described across settings in the catalog introduction; this OE_TO_maths_en_COMP / train selection expects 200 task records with a source-row limit of 200.","answer_access":"Fields outside the question allowlist, including answers and solutions when supplied, remain in restricted storage; the converter alone does not establish whether usable answers are available.","license":"apache-2.0 (declared on the Hugging Face dataset card)","history":"The OlympiadBench research team introduced the benchmark in 2024 to evaluate scientific reasoning on difficult mathematics and physics problems. Its official repository is hosted by OpenBMB, and the Hothan dataset distribution on Hugging Face was published in July 2024. The benchmark includes English and Chinese material and both text-only and multimodal settings.","construction":"OlympiadBench draws on existing Olympiad and examination problems in mathematics and physics. The official description lists 8,476 problems across its English and Chinese settings, including tasks with visual content. The catalog's selection contains 200 records from an English text-only mathematics competition setting, rather than a sample covering every setting. Records distinguish properties such as answer type, units and whether multiple answers are expected.","limitations":"An OlympiadBench score needs to identify the evaluated language, subject and text-only or multimodal setting. Performance on English text-only mathematics cannot be treated as evidence of equal ability on Chinese problems or visual physics tasks. Because the benchmark draws on existing examination material, training-data overlap is a potential concern rather than something a high score can rule out. The tasks assess solving posed problems, not open-ended scientific investigation.","saturation":"","citation":"@article{he2024olympiadbench,\n  title={Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems},\n  author={He, Chaoqun and Luo, Renjie and Bai, Yuzhuo and Hu, Shengding and Thai, Zhen Leng and Shen, Junhao and Hu, Jinyi and Han, Xu and Huang, Yujie and Zhang, Yuxiang and others},\n  journal={arXiv preprint arXiv:2402.14008},\n  year={2024}\n}","maintainer":"Hothan (Hugging Face organisation)","released":"Dataset published on Hugging Face 2024-07","why_hard":"Requires scientific reasoning on Olympiad-level mathematics and physics problems.","related":["hle","critpt","frontierscience"],"scores":[]}},"facts":{"slug":"olympiadbench","task_count":200,"image_count":0,"formats":[{"count":200,"value":"text"}],"answers":[{"count":200,"value":"published"}],"configs":[{"count":200,"value":"OE_TO_maths_en_COMP"}],"splits":[{"count":200,"value":"train"}],"waiting_count":200,"post_count":0,"discussed_count":0,"releases":[{"acquired":"2026-09-09","revision":"91184b52131e7fc9455fef848035173aea8cc01a","release_id":"54350c3251caf7e2b81eb1a8412020dcc52511e8a7974ba46239baf26d007e72"},{"acquired":"2026-09-09","revision":"91184b52131e7fc9455fef848035173aea8cc01a","release_id":"e6b5c6cafc50c51d4d2d7d3ffe5c91385923d5da675c005e8dbfd77e93a1b4d3"},{"acquired":"2026-09-09","revision":"91184b52131e7fc9455fef848035173aea8cc01a","release_id":"479001f7f1d0a3cc2995fd8aff223e1c819e80c144cd17bb62aab2095ddcb602"},{"acquired":"2026-09-09","revision":"91184b52131e7fc9455fef848035173aea8cc01a","release_id":"f67d450557c07dd61615f9d7907996f92d7893e2f95e213adc1d82ef5bfc72ed"}],"terms":{"license":null,"checked_at":null,"distribution_terms":null,"publisher_requests":null}}},{"benchmark":{"slug":"mmmu","formal_name":"MMMU","category":"Multimodal","effective_mode":"metadata_only","profile":{"resources":[{"kind":"dataset","url":"https://huggingface.co/datasets/MMMU/MMMU","label":"Source dataset"},{"kind":"issues","url":"https://huggingface.co/datasets/MMMU/MMMU/discussions","label":"Dataset discussions"},{"kind":"site","url":"https://mmmu-benchmark.github.io/","label":"Official site"},{"kind":"paper","url":"https://arxiv.org/abs/2311.16502","label":"arXiv:2311.16502"}],"task_format":"A question with several images or diagrams and a set of options.","scoring":"The source ships the answer; nothing is imported here.","metric":"","size":"11,500 questions in the source; none imported.","answer_access":"No problem text is imported, so there is no answer storage for this entry. The source distributes the answers under its own terms.","license":"apache-2.0 (declared on the Hugging Face dataset card)","history":"The MMMU research team introduced the benchmark in 2023 to assess multimodal understanding and reasoning at the level of college subject knowledge. Its source dataset was published under the MMMU organisation on Hugging Face in November 2023. The benchmark aimed to extend evaluation beyond everyday image recognition to questions requiring specialist knowledge and interpretation of academic visual material.","construction":"MMMU collects 11,500 college-level questions across six disciplines and 30 subjects. Questions were gathered from educational materials and combine text with visual content such as diagrams, charts and other images. Its subject coverage makes the evaluation a mixture of domain knowledge and interpretation of visual evidence, rather than a collection of interchangeable image-recognition tasks.","limitations":"An aggregate MMMU score combines specialist knowledge, visual interpretation and answer selection, which makes it difficult to attribute a failure to any one capability. Performance can also vary substantially between subjects, so the overall number can obscure uneven coverage. Multiple-choice items allow correct selections without demonstrating a complete explanation. MMMU-Pro is a revised benchmark and its results should not be treated as scores on the original MMMU set.","saturation":"","citation":"@article{yue2023mmmu,\n  title={MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI},\n  author={Xiang Yue and Yuansheng Ni and Kai Zhang and Tianyu Zheng and Ruoqi Liu and Ge Zhang and Samuel Stevens and Dongfu Jiang and Weiming Ren and Yuxuan Sun and Cong Wei and Botao Yu and Ruibin Yuan and Renliang Sun and Ming Yin and Boyuan Zheng and Zhenzhu Yang and Yibo Liu and Wenhao Huang and Huan Sun and Yu Su and Wenhu Chen},\n  journal={arXiv preprint arXiv:2311.16502},\n  year={2023}\n}","maintainer":"MMMU (source organisation)","released":"Dataset published on Hugging Face 2023-11","why_hard":"The figures carry information the text does not repeat, so a model that reads only the question is answering a different, easier question.","related":["mmmu-pro","mathvista","hle"],"scores":[]}},"facts":null},{"benchmark":{"slug":"mmmu-pro","formal_name":"MMMU-Pro","category":"Multimodal","effective_mode":"full","profile":{"resources":[{"kind":"dataset","url":"https://huggingface.co/datasets/MMMU/MMMU_Pro","label":"Source dataset"},{"kind":"issues","url":"https://huggingface.co/datasets/MMMU/MMMU_Pro/discussions","label":"Dataset discussions"},{"kind":"site","url":"https://mmmu-benchmark.github.io/","label":"Official site"},{"kind":"paper","url":"https://arxiv.org/abs/2409.02813","label":"arXiv:2409.02813"}],"task_format":"Image-and-text questions. The selected standard (10 options) import accepts scalar question, options, subject, and img_type fields, and extracts supplied image_1 through image_7 as image assets.","scoring":"For the 200 imported problems, the source ships an answer with every problem; this catalogue keeps it in restricted storage and never renders it. Questions are multiple choice, including a vision-only setting where the question is embedded in a screenshot.","metric":"accuracy over multiple-choice questions (dataset card)","size":"1,730 questions per setting described in the catalog introduction; this standard (10 options) / test selection expects 200 task records with a source-row limit of 200.","answer_access":"Answer and explanation fields, when supplied, remain in restricted storage, separate from question fields and image assets.","license":"apache-2.0 (declared on the Hugging Face dataset card)","history":"The MMMU research team released MMMU-Pro in 2024 as a more demanding evaluation of multimodal understanding across academic disciplines. It extends the MMMU evaluation family with distinct standard and vision settings. The dataset recorded here was published by the MMMU organization on Hugging Face in August 2024.","construction":"","limitations":"Multiple-choice accuracy measures answer selection rather than the quality of a free-form explanation or the ability to carry out an academic workflow. The official card distinguishes four-option and ten-option standard settings from a vision setting in which the question is embedded in a screenshot; these are different evaluation conditions. Scores from these settings should not be pooled or compared as though their input and answer spaces were identical. The catalog's 200-task import from the ten-option setting is also smaller than the 1,730 questions per setting described by the official card.","saturation":"","citation":"@article{yue2024mmmu,\n  title={MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark},\n  author={Xiang Yue and Tianyu Zheng and Yuansheng Ni and Yubo Wang and Kai Zhang and Shengbang Tong and Yuxuan Sun and Botao Yu and Ge Zhang and Huan Sun and Yu Su and Wenhu Chen and Graham Neubig},\n  journal={arXiv preprint arXiv:2409.02813},\n  year={2024}\n}","maintainer":"MMMU (Hugging Face organisation)","released":"Dataset published on Hugging Face 2024-08","why_hard":"Requires multimodal understanding across academic disciplines using images and text.","related":["hle","gaia","olympiadbench"],"scores":[]}},"facts":{"slug":"mmmu-pro","task_count":200,"image_count":240,"formats":[{"count":200,"value":"text"}],"answers":[{"count":200,"value":"published"}],"configs":[{"count":200,"value":"standard (10 options)"}],"splits":[{"count":200,"value":"test"}],"waiting_count":200,"post_count":0,"discussed_count":0,"releases":[{"acquired":"2026-09-09","revision":"563f3e84bb3b90893083a1f039cfa13077f2302b","release_id":"cabe427ffc667207714dc67ea0604e33d9d6f0092f190318841999121d00a3ac"},{"acquired":"2026-09-09","revision":"563f3e84bb3b90893083a1f039cfa13077f2302b","release_id":"90825fa194d8801c700b00f9479a3471f3a1ce0ed838cde8f7a82c4a630d377e"},{"acquired":"2026-09-09","revision":"563f3e84bb3b90893083a1f039cfa13077f2302b","release_id":"ff4a0fbfe387bb08b1ac47e7bb2a02f221f3e1e5ca4f0d74d37ad3c4e4fd22ef"},{"acquired":"2026-09-09","revision":"563f3e84bb3b90893083a1f039cfa13077f2302b","release_id":"c40283c074e64a67543bc83f78393ef6bf39ff50d75c9fb83cc6ed6d0ca0d222"}],"terms":{"license":null,"checked_at":null,"distribution_terms":null,"publisher_requests":null}}},{"benchmark":{"slug":"arc-agi-2","formal_name":"ARC-AGI-2","category":"Reasoning","effective_mode":"full","profile":{"resources":[{"kind":"repository","url":"https://github.com/arcprize/ARC-AGI-2","label":"Source repository"},{"kind":"issues","url":"https://github.com/arcprize/ARC-AGI-2/issues","label":"Repository issues"}],"task_format":"Each task contains train input/output grid pairs and test input grids. Grids are rectangular, 1–30 rows and columns, with integer colors 0–9.","scoring":"For the 120 imported problems, the source ships an answer with every problem; this catalogue keeps it in restricted storage and never renders it.","metric":"","size":"120 public evaluation tasks in this catalog selection; training is a separate split.","answer_access":"Training outputs are examples in the prompt. Test output grids, when supplied, are separated into restricted answers; missing test outputs are allowed.","license":"Apache-2.0 (declared in the GitHub repository)","history":"ARC Prize introduced ARC-AGI-2 in 2025 to test whether systems can infer unfamiliar rules from a small number of demonstrations. It continued the ARC approach of using abstract grid transformations to study generalization beyond familiar task templates; the source repository was created in March 2025.","construction":"Tasks consist of demonstration input-output grid pairs followed by test inputs requiring the same underlying transformation. Grids use ten integer color values and range from one to thirty rows and columns. The catalogue selection contains 120 public evaluation tasks, with training tasks in a separate split and supplied test outputs stored separately from prompts.","limitations":"Success concerns a constrained world of colored grids and does not establish competence in language, physical interaction, or extended real-world tasks. The public evaluation tasks include accessible target outputs in the source, so restricting answers in this catalogue does not make the underlying benchmark private. Results on these public tasks should therefore be distinguished from results on held-out evaluations.","saturation":"","citation":"","maintainer":"arcprize (GitHub organisation)","released":"Repository created on GitHub 2025-03","why_hard":"Requires inferring unfamiliar rules from a few examples and applying them to new grids.","related":["hle","bbeh","longbench-v2"],"scores":[]}},"facts":{"slug":"arc-agi-2","task_count":120,"image_count":0,"formats":[{"count":120,"value":"grid"}],"answers":[{"count":120,"value":"published"}],"configs":[{"count":120,"value":""}],"splits":[{"count":120,"value":"evaluation"}],"waiting_count":119,"post_count":1,"discussed_count":1,"releases":[{"acquired":"2026-09-09","revision":"f3283f727488ad98fe575ea6a5ac981e4a188e49","release_id":"d292864f8da6f80827fc0bcb1e21f7ec47b0862b561fe3a899ba85a5ac0498c6"},{"acquired":"2026-09-09","revision":"f3283f727488ad98fe575ea6a5ac981e4a188e49","release_id":"e20b468c97b2aa66231f95b42ab60cfcf646cf87f5f6c35c20c407d73170b712"},{"acquired":"2026-09-09","revision":"f3283f727488ad98fe575ea6a5ac981e4a188e49","release_id":"199f8507c15a154ce31fab78a68bef7e1ce640ea98adbb31098808a10fce4bda"},{"acquired":"2026-09-09","revision":"f3283f727488ad98fe575ea6a5ac981e4a188e49","release_id":"69480bfd3a146900e7a5d2a669c69e8e5c9eb9b3d0f10c9a18c51b00933f0a0e"},{"acquired":"2026-09-08","revision":"f3283f727488ad98fe575ea6a5ac981e4a188e49","release_id":"dce142b1444c8afbc4f8279e289d4e1f58946fa8ad2d6c49b41c7c705db45394"}],"terms":{"license":null,"checked_at":null,"distribution_terms":null,"publisher_requests":null}}},{"benchmark":{"slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","category":"Reasoning","effective_mode":"full","profile":{"resources":[{"kind":"repository","url":"https://github.com/google-deepmind/bbeh","label":"Source repository"},{"kind":"issues","url":"https://github.com/google-deepmind/bbeh/issues","label":"Repository issues"}],"task_format":"Text prompts in the input field of examples; the selection is the full distribution across all 23 task categories.","scoring":"The source ships an answer with every problem; this catalogue keeps it in restricted storage and never renders it. The repository ships the official evaluation code as bbeh/evaluate.py.","metric":"","size":"Catalog selection: the full distribution, 4,513 unique questions across all 23 task directories; 7 source rows repeat an earlier prompt and are recorded as variants. The mini distribution repeats a subset and is not imported separately.","answer_access":"The target field, when supplied, is separated from input into restricted answer storage.","license":"Apache-2.0 (declared in the GitHub repository)","history":"Google DeepMind developed BIG-Bench Extra Hard as a more demanding evaluation of language-model reasoning following BIG-Bench Hard. The source repository was created in February 2025. The benchmark uses challenging task categories to probe reasoning beyond the difficulty of its predecessor.","construction":"The repository distributes full and mini versions with examples represented as text inputs and target answers, alongside official evaluation code. This catalogue selects 200 Boolean-expression questions from the full distribution. Those records represent one category and should not be interpreted as the size or composition of the complete benchmark.","limitations":"Performance on the Boolean-expression selection measures a narrower capability than performance across the complete suite. An aggregate reported for BIG-Bench Extra Hard therefore cannot be treated as an accuracy figure for these 200 records. Comparisons also need to distinguish the full and mini distributions and use consistent answer-evaluation rules.","saturation":"","citation":"@article{bbeh,\n      title={BIG-Bench Extra Hard},\n      author={Mehran Kazemi, Bahare Fatemi, Hritik Bansal, John Palowitch, Chrysovalantis Anastasiou, Sanket Vaibhav Mehta, Lalit K. Jain, Virginia Aglietti, Disha Jindal, Peter Chen, Nishanth Dikkala, Gladys Tyen, Xin Liu, Uri Shalit, Silvia Chiappa, Kate Olszewska, Yi Tay, Vinh Q. Tran, Quoc V. Le, Orhan Firat},\n      journal={arXiv preprint arXiv:2502.19187},\n      year={2025},\n}","maintainer":"google-deepmind (GitHub organisation)","released":"Repository created on GitHub 2025-02","why_hard":"Requires reasoning over challenging tasks; this selection focuses on Boolean expressions.","related":["hle","arc-agi-2","longbench-v2"],"scores":[]}},"facts":{"slug":"bbeh","task_count":4513,"image_count":0,"formats":[{"count":4513,"value":"text"}],"answers":[{"count":4511,"value":"published"},{"count":2,"value":"unavailable"}],"configs":[{"count":200,"value":"bbeh_boardgame_qa"},{"count":200,"value":"bbeh_boolean_expressions"},{"count":200,"value":"bbeh_buggy_tables"},{"count":200,"value":"bbeh_causal_understanding"},{"count":120,"value":"bbeh_disambiguation_qa"},{"count":200,"value":"bbeh_dyck_languages"},{"count":200,"value":"bbeh_geometric_shapes"},{"count":200,"value":"bbeh_hyperbaton"},{"count":194,"value":"bbeh_linguini"},{"count":200,"value":"bbeh_movie_recommendation"},{"count":200,"value":"bbeh_multistep_arithmetic"},{"count":200,"value":"bbeh_nycc"},{"count":200,"value":"bbeh_object_counting"},{"count":200,"value":"bbeh_object_properties"},{"count":200,"value":"bbeh_sarc_triples"},{"count":200,"value":"bbeh_shuffled_objects"},{"count":199,"value":"bbeh_spatial_reasoning"},{"count":200,"value":"bbeh_sportqa"},{"count":200,"value":"bbeh_temporal_sequence"},{"count":200,"value":"bbeh_time_arithmetic"},{"count":200,"value":"bbeh_web_of_lies"},{"count":200,"value":"bbeh_word_sorting"},{"count":200,"value":"bbeh_zebra_puzzles"}],"splits":[{"count":4513,"value":"full"}],"waiting_count":4513,"post_count":0,"discussed_count":0,"releases":[{"acquired":"2026-09-23","revision":"80d12ca916b7158f22293fcf3144f4d3d854d4be","release_id":"7da5c6a85d8ddf2e575c4a99a35d793e73ef5000561cc33277b4a6ba430a457a"},{"acquired":"2026-09-09","revision":"80d12ca916b7158f22293fcf3144f4d3d854d4be","release_id":"dfbc3f2ad31e6b16bc142150ffc363a51bb6ebddec99930b2625e82c332c8951"},{"acquired":"2026-09-09","revision":"80d12ca916b7158f22293fcf3144f4d3d854d4be","release_id":"5971a9aa3df4345d93e26deda1b8b0e0be0f3870afb1838146ed72cde4f8ee4b"},{"acquired":"2026-09-09","revision":"80d12ca916b7158f22293fcf3144f4d3d854d4be","release_id":"22a6c6a76f02deb05ff1f7aff4d83df0738c245295339ebca4130b3473b9bcf9"},{"acquired":"2026-09-08","revision":"80d12ca916b7158f22293fcf3144f4d3d854d4be","release_id":"27dd7d5361f8a594c618a3774ec9cd888568a6db1aa7324028c10a85e9024ab1"}],"terms":{"license":null,"checked_at":null,"distribution_terms":null,"publisher_requests":null}}},{"benchmark":{"slug":"bbh","formal_name":"BIG-Bench Hard","category":"Reasoning","effective_mode":"full","profile":{"resources":[{"kind":"repository","url":"https://github.com/suzgunmirac/BIG-Bench-Hard","label":"Official repository"},{"kind":"paper","url":"https://arxiv.org/abs/2210.09261","label":"arXiv:2210.09261"}],"task_format":"Varies by task: boolean expressions, date understanding, object counting, logical deduction and others.","scoring":"Each task ships its own target with every problem; this catalogue keeps it in restricted storage and never renders it. The repository ships the official evaluation code.","metric":"","size":"6,507 unique problems imported from the 27 task files of the official repository (6,511 rows; 4 rows repeat an earlier prompt and are recorded as variants).","answer_access":"The target field of every example is separated from the input into restricted answer storage and never rendered.","license":"mit (declared on the Hugging Face dataset card)","history":"BIG-Bench Hard was introduced in 2022 by Mirac Suzgun and collaborators to examine reasoning tasks that remained difficult for language models despite increased scale. The group selected 23 BIG-Bench tasks on which the evaluated models fell below average human performance. Their study examined how chain-of-thought prompting changed performance on this difficult subset.","construction":"BBH reuses a selected subset of BIG-Bench rather than collecting a uniform new set of questions. Its tasks include boolean expressions, date understanding, object counting, and logical deduction, with task-specific targets and answer formats. The source described in the catalogue contains about 6,500 examples across the tasks.","limitations":"The hard designation reflects the models and evaluation conditions used for the original selection, rather than a permanent difficulty threshold. Task formats differ, so an aggregate score can conceal uneven abilities across the 23 tasks. Prompt examples, chain-of-thought instructions, and final-answer extraction affect comparability, while the public questions and targets create a potential route for training-data overlap.","saturation":"The catalogue records a vendor-reported exact-match score of 87.5 for the DeepSeek-V3 base model with three-shot prompting. This result leaves errors under that setup, but it is neither a complete leaderboard nor evidence of the current performance ceiling. Per-task results and matched prompting conditions remain necessary to determine whether a particular part of BBH still separates systems.","citation":"@article{suzgun2022bbh,\n  title={Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them},\n  author={Mirac Suzgun and Nathan Scales and Nathanael Schärli and Sebastian Gehrmann and Yi Tay and Hyung Won Chung and Aakanksha Chowdhery and Quoc V. Le and Ed H. Chi and Denny Zhou and Jason Wei},\n  journal={arXiv preprint arXiv:2210.09261},\n  year={2022}\n}","maintainer":"maveriq (source organisation)","released":"Dataset published on Hugging Face 2023-09","why_hard":"These are exactly the tasks that resisted scale, and several of them need an explicit multi-step procedure rather than a single judgement.","related":["bbeh","mmlu","musr"],"scores":[{"model":"DeepSeek-V3","organization":"DeepSeek","score":87.5,"metric":"EM","setup":"Base model; 3-shot.","reported_by":"vendor","source_url":"https://raw.githubusercontent.com/deepseek-ai/DeepSeek-V3/main/README.md","source_label":"Base Model, Standard Benchmarks, BBH (EM), final column","reported_at":"","checked_at":"2026-09-16"}]}},"facts":{"slug":"bbh","task_count":6507,"image_count":0,"formats":[{"count":6507,"value":"text"}],"answers":[{"count":6507,"value":"published"}],"configs":[{"count":250,"value":"boolean_expressions"},{"count":185,"value":"causal_judgement"},{"count":250,"value":"date_understanding"},{"count":250,"value":"disambiguation_qa"},{"count":250,"value":"dyck_languages"},{"count":250,"value":"formal_fallacies"},{"count":250,"value":"geometric_shapes"},{"count":250,"value":"hyperbaton"},{"count":250,"value":"logical_deduction_five_objects"},{"count":250,"value":"logical_deduction_seven_objects"},{"count":250,"value":"logical_deduction_three_objects"},{"count":250,"value":"movie_recommendation"},{"count":250,"value":"multistep_arithmetic_two"},{"count":250,"value":"navigate"},{"count":250,"value":"object_counting"},{"count":146,"value":"penguins_in_a_table"},{"count":250,"value":"reasoning_about_colored_objects"},{"count":250,"value":"ruin_names"},{"count":250,"value":"salient_translation_error_detection"},{"count":178,"value":"snarks"},{"count":248,"value":"sports_understanding"},{"count":250,"value":"temporal_sequences"},{"count":250,"value":"tracking_shuffled_objects_five_objects"},{"count":250,"value":"tracking_shuffled_objects_seven_objects"},{"count":250,"value":"tracking_shuffled_objects_three_objects"},{"count":250,"value":"web_of_lies"},{"count":250,"value":"word_sorting"}],"splits":[{"count":6507,"value":"main"}],"waiting_count":6507,"post_count":0,"discussed_count":0,"releases":[{"acquired":"2026-09-23","revision":"9ee07bd481feebf959a6b59d61ea57bdcf30964d","release_id":"af4232583defeeda8b3b26a671268297bce1e0f88fce6e89245d47140d859964"},{"acquired":null,"revision":"catalog-20-381f44295965a2e9","release_id":null}],"terms":{"license":null,"checked_at":null,"distribution_terms":null,"publisher_requests":null}}},{"benchmark":{"slug":"boolq","formal_name":"BoolQ","category":"Reasoning","effective_mode":"full","profile":{"resources":[{"kind":"dataset","url":"https://huggingface.co/datasets/google/boolq","label":"Source dataset"},{"kind":"issues","url":"https://huggingface.co/datasets/google/boolq/discussions","label":"Dataset discussions"},{"kind":"repository","url":"https://github.com/google-research-datasets/boolean-questions","label":"Official repository"},{"kind":"paper","url":"https://arxiv.org/abs/1905.10044","label":"arXiv:1905.10044"}],"task_format":"A yes/no question and the passage it is asked against. The boolean answer is not part of the prompt.","scoring":"The source ships the boolean answer with every question; this catalogue keeps it in restricted storage and never renders it.","metric":"accuracy over the 3,270 validation questions","size":"3,270 problems imported from the default / validation selection (the full validation split).","answer_access":"Columns outside the prompt allowlist, including any answer or reference solution, are separated into restricted storage and never rendered.","license":"cc-by-sa-3.0 (declared on the Hugging Face dataset card)","history":"Google researchers introduced BoolQ in 2019 to study yes-or-no questions that arise naturally, rather than questions written specifically for a reading-comprehension test. The motivation was to test whether systems could infer an answer from a passage when the question did not simply repeat its wording. The March 2022 Hugging Face publication date recorded for this source refers to the hosted dataset, not the original research release.","construction":"BoolQ pairs naturally occurring questions from search queries with passages that provide evidence for a boolean answer. Human annotation establishes whether the passage supports a yes or no response. The source validation split contains 3,270 questions, of which this catalogue imports 200; accuracy over the full validation split is distinct from accuracy over that smaller selection.","limitations":"Binary accuracy measures the chosen label without checking whether a model identified the supporting evidence or used valid reasoning. Since the relevant passage is supplied, BoolQ does not measure retrieval or the ability to find an authoritative source. Performance on a fixed public validation set also cannot by itself establish generalization to unseen search questions or rule out prior exposure to the examples.","saturation":"","citation":"@article{clark2019boolq,\n  title={BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions},\n  author={Christopher Clark and Kenton Lee and Ming-Wei Chang and Tom Kwiatkowski and Michael Collins and Kristina Toutanova},\n  journal={arXiv preprint arXiv:1905.10044},\n  year={2019}\n}","maintainer":"google (source organisation)","released":"Dataset published on Hugging Face 2022-03","why_hard":"The questions were written by people who wanted to know the answer, not to make a test, so they rarely line up with a single sentence of the passage.","related":["drop","mmlu","hellaswag"],"scores":[]}},"facts":{"slug":"boolq","task_count":3270,"image_count":0,"formats":[{"count":3270,"value":"text"}],"answers":[{"count":3270,"value":"published"}],"configs":[{"count":3270,"value":"default"}],"splits":[{"count":3270,"value":"validation"}],"waiting_count":3270,"post_count":0,"discussed_count":0,"releases":[{"acquired":"2026-09-12","revision":"35b264d03638db9f4ce671b711558bf7ff0f80d5","release_id":"9192c36c5220f4842f936ce4cec538054dc9de447d5fdc1c38e5fc8da397606d"},{"acquired":"2026-09-12","revision":"35b264d03638db9f4ce671b711558bf7ff0f80d5","release_id":"fc6c571be9dd817b6c72cef00fc4e9e4d923aaabeac43b7d1ee00de7a688b950"},{"acquired":"2026-09-12","revision":"35b264d03638db9f4ce671b711558bf7ff0f80d5","release_id":"56068dc8733a8f705dc88c0f3c59865407bcefd6f802957c00a91a895d95f77c"}],"terms":{"license":null,"checked_at":null,"distribution_terms":null,"publisher_requests":null}}},{"benchmark":{"slug":"drop","formal_name":"DROP","category":"Reasoning","effective_mode":"full","profile":{"resources":[{"kind":"dataset","url":"https://huggingface.co/datasets/ucinlp/drop","label":"Source dataset"},{"kind":"issues","url":"https://huggingface.co/datasets/ucinlp/drop/discussions","label":"Dataset discussions"},{"kind":"paper","url":"https://arxiv.org/abs/1903.00161","label":"arXiv:1903.00161"}],"task_format":"A passage and a question about it. The answer spans and numbers are not part of the prompt.","scoring":"The source ships answer spans with every question; this catalogue keeps them in restricted storage and never renders them.","metric":"exact match and macro-averaged F1 over the answer spans","size":"9,535 problems imported from the default / validation selection (the full validation split).","answer_access":"Columns outside the prompt allowlist, including any answer or reference solution, are separated into restricted storage and never rendered.","license":"cc-by-sa-4.0 (declared on the Hugging Face dataset card)","history":"The Allen Institute for AI and collaborating researchers introduced DROP in 2019 to test reading comprehension that requires discrete reasoning over a passage. It targeted operations such as counting, comparison and arithmetic that were poorly captured by benchmarks focused on locating an answer span. The March 2022 publication date for this source describes the ucinlp Hugging Face distribution rather than the original benchmark release.","construction":"DROP was built from Wikipedia passages, with crowdworkers writing questions that require reasoning over the information they contain. Its collection process used an existing question-answering model as an adversary to discourage questions that the model could already answer. Answers can be numbers, dates or text spans, and the source validation split contains 9,535 questions. Evaluation uses exact match and an answer-level F1 measure.","limitations":"DROP combines language understanding with discrete operations, so an incorrect answer does not isolate whether the failure came from reading, selecting facts or calculation. It includes span answers as well as numerical answers; it is not a pure arithmetic test. Exact match and F1 grade the final answer rather than the derivation, and partial overlap credit should not be read as proof of a partially correct reasoning process.","saturation":"The vendor report recorded here gives the DeepSeek-V3 base model an F1 score of 89.0 in a three-shot setup. This is high on the reported metric but still below its ceiling, and it does not imply that 89 percent of questions were answered exactly correctly. That single result is insufficient to establish current saturation or the performance of systems using other prompting and evaluation settings.","citation":"@article{dua2019drop,\n  title={DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs},\n  author={Dheeru Dua and Yizhong Wang and Pradeep Dasigi and Gabriel Stanovsky and Sameer Singh and Matt Gardner},\n  journal={arXiv preprint arXiv:1903.00161},\n  year={2019}\n}","maintainer":"ucinlp (source organisation)","released":"Dataset published on Hugging Face 2022-03","why_hard":"The passage contains the facts but not the answer. Getting it right means locating several numbers, deciding which operation relates them, and carrying it out exactly.","related":["longbench-v2","musr","boolq"],"scores":[{"model":"DeepSeek-V3","organization":"DeepSeek","score":89,"metric":"F1","setup":"Base model; 3-shot.","reported_by":"vendor","source_url":"https://raw.githubusercontent.com/deepseek-ai/DeepSeek-V3/main/README.md","source_label":"Base Model, Standard Benchmarks, DROP (F1), final column","reported_at":"","checked_at":"2026-09-16"}]}},"facts":{"slug":"drop","task_count":9535,"image_count":0,"formats":[{"count":9535,"value":"text"}],"answers":[{"count":9535,"value":"published"}],"configs":[{"count":9535,"value":"default"}],"splits":[{"count":9535,"value":"validation"}],"waiting_count":9535,"post_count":0,"discussed_count":0,"releases":[{"acquired":"2026-09-12","revision":"95cda593fae71b60b5b19f82de3fcf3298c1239c","release_id":"c2d8b15a0e8959695eb4b3faae285215eb9032f7c8f7380876596f54d069f4f6"},{"acquired":"2026-09-12","revision":"95cda593fae71b60b5b19f82de3fcf3298c1239c","release_id":"974819cfe356e452c09316646733173d18e4546e713d9c90ea8a70fa1eaffcf7"},{"acquired":"2026-09-12","revision":"95cda593fae71b60b5b19f82de3fcf3298c1239c","release_id":"fc596d0216e72616a5ffb7973a42a4e7623cc55fefd30b26b302dab4caf0aae7"}],"terms":{"license":null,"checked_at":null,"distribution_terms":null,"publisher_requests":null}}},{"benchmark":{"slug":"hellaswag","formal_name":"HellaSwag","category":"Reasoning","effective_mode":"full","profile":{"resources":[{"kind":"dataset","url":"https://huggingface.co/datasets/Rowan/hellaswag","label":"Source dataset"},{"kind":"issues","url":"https://huggingface.co/datasets/Rowan/hellaswag/discussions","label":"Dataset discussions"},{"kind":"site","url":"https://rowanzellers.com/hellaswag/","label":"Official site"},{"kind":"paper","url":"https://arxiv.org/abs/1905.07830","label":"arXiv:1905.07830"}],"task_format":"An activity label and a context sentence, with the four candidate continuations as a list. Which one is correct is not part of the prompt.","scoring":"The source ships the index of the correct continuation; this catalogue keeps it in restricted storage and never renders it.","metric":"accuracy over the 10,042 validation items","size":"10,042 problems imported from the default / validation selection (the full validation split).","answer_access":"Columns outside the prompt allowlist, including any answer or reference solution, are separated into restricted storage and never rendered.","license":"mit (declared in the dataset card’s Licensing Information section, which points at the project’s MIT LICENSE)","history":"Rowan Zellers and collaborators at the University of Washington and the Allen Institute for AI introduced HellaSwag in 2019 to test commonsense inference about everyday situations. They sought examples that people could resolve easily but that language models could not solve reliably through surface plausibility alone. The March 2022 Hugging Face publication date refers to the hosted dataset distribution, not the original benchmark release.","construction":"HellaSwag draws contexts and human-written continuations from ActivityNet video captions and WikiHow descriptions of activities. Language models generate candidate incorrect endings, and adversarial filtering selects distractors that are difficult for discriminative models to reject. Human validation checks the resulting multiple-choice examples. Each item offers four continuations; the validation split contains 10,042 items, of which this catalogue imports 200.","limitations":"The task tests selection among supplied endings rather than the ability to generate an appropriate continuation or act in the described situation. Its adversarial filtering reflects the models used during construction and does not guarantee equal difficulty for later systems. The public dataset presents a potential contamination risk, while differences in prompting and candidate-scoring procedures can complicate comparisons between reports.","saturation":"The vendor result recorded here reports 88.9% accuracy for the DeepSeek-V3 base model with 10-shot prompting. That is high performance but remains below perfect accuracy on the reported evaluation. This single result does not establish that HellaSwag is saturated; comparisons near the upper end need to keep the split and evaluation procedure consistent.","citation":"@article{zellers2019hellaswag,\n  title={HellaSwag: Can a Machine Really Finish Your Sentence?},\n  author={Rowan Zellers and Ari Holtzman and Yonatan Bisk and Ali Farhadi and Yejin Choi},\n  journal={arXiv preprint arXiv:1905.07830},\n  year={2019}\n}","maintainer":"Rowan (source organisation)","released":"Dataset published on Hugging Face 2022-03","why_hard":"The distractors were selected specifically because models rank them highly, so the gap it measures is between surface plausibility and actual commonsense.","related":["winogrande","arc-challenge","mmlu"],"scores":[{"model":"DeepSeek-V3","organization":"DeepSeek","score":88.9,"metric":"Acc.","setup":"Base model; 10-shot.","reported_by":"vendor","source_url":"https://raw.githubusercontent.com/deepseek-ai/DeepSeek-V3/main/README.md","source_label":"Base Model, Standard Benchmarks, HellaSwag (Acc.), final column","reported_at":"","checked_at":"2026-09-16"}]}},"facts":{"slug":"hellaswag","task_count":10042,"image_count":0,"formats":[{"count":10042,"value":"text"}],"answers":[{"count":10042,"value":"published"}],"configs":[{"count":10042,"value":"default"}],"splits":[{"count":10042,"value":"validation"}],"waiting_count":10042,"post_count":0,"discussed_count":0,"releases":[{"acquired":"2026-09-12","revision":"218ec52e09a7e7462a5400043bb9a69a41d06b76","release_id":"7e083cabda9ab710ca6098d0a7847c0298da5173cb8992815fffd936fd3418ec"},{"acquired":"2026-09-12","revision":"218ec52e09a7e7462a5400043bb9a69a41d06b76","release_id":"da7444a359894e425a61cf67785afb53ae36b58602462ebe5831022f2bf11535"},{"acquired":"2026-09-12","revision":"218ec52e09a7e7462a5400043bb9a69a41d06b76","release_id":"7fb6311c6df260c1338608f05b7793ad96d17a14c0d611ea25ec36462195a343"}],"terms":{"license":null,"checked_at":null,"distribution_terms":null,"publisher_requests":null}}},{"benchmark":{"slug":"mmlu-pro","formal_name":"MMLU-Pro","category":"Reasoning","effective_mode":"full","profile":{"resources":[{"kind":"dataset","url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","label":"Source dataset"},{"kind":"issues","url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro/discussions","label":"Dataset discussions"},{"kind":"paper","url":"https://arxiv.org/abs/2406.01574","label":"arXiv:2406.01574"}],"task_format":"Multiple-choice questions with ten options. The import accepts question, options, category and src.","scoring":"The source ships an answer with every problem; this catalogue keeps it in restricted storage and never renders it.","metric":"","size":"12,032 problems imported from the default / test selection (the full test split).","answer_access":"Columns outside the prompt allowlist, including any answer or reference solution, are separated into restricted storage and never rendered.","license":"mit (declared on the source)","history":"TIGER-Lab introduced MMLU-Pro in 2024 to make broad academic question answering more demanding as the original MMLU became less discriminating. The revision emphasised harder reasoning questions and expanded the answer choices from four to ten. Its Hugging Face publication is recorded as May 2024.","construction":"MMLU-Pro rebuilds the question collection by removing easier original items and using expanded sets of answer options. It covers fourteen academic and professional subjects in a multiple-choice format. The catalogue imports 200 test questions, a capped selection that should not be treated as the size of the complete benchmark.","limitations":"Ten options reduce the probability of uniform random guessing but do not prevent memorisation or exploitation of answer cues. A correct choice does not directly validate a model's explanation or its ability to solve an open-ended version of the task. Subject composition, prompting and answer extraction remain relevant to comparisons, and reuse of public question material leaves a risk of training exposure.","saturation":"The recorded vendor report gives DeepSeek-V3 base 64.4 percent accuracy with five examples in the prompt. That result remains well below the ceiling for the reported setup and supports treating the test as challenging for that model. It does not establish the current leading score or the remaining headroom for every model family.","citation":"","maintainer":"TIGER-Lab (source organisation)","released":"Dataset published on Hugging Face 2024-05","why_hard":"Ten options rather than four, and the easiest original items removed, so guessing and memorisation carry much less of the score.","related":["arc-agi-2","arc-agi-3","bbeh"],"scores":[{"model":"DeepSeek-V3","organization":"DeepSeek","score":64.4,"metric":"Acc.","setup":"Base model; 5-shot.","reported_by":"vendor","source_url":"https://raw.githubusercontent.com/deepseek-ai/DeepSeek-V3/main/README.md","source_label":"Base Model, Standard Benchmarks, MMLU-Pro (Acc.), final column","reported_at":"","checked_at":"2026-09-16"}]}},"facts":{"slug":"mmlu-pro","task_count":12032,"image_count":0,"formats":[{"count":12032,"value":"text"}],"answers":[{"count":12032,"value":"published"}],"configs":[{"count":12032,"value":"default"}],"splits":[{"count":12032,"value":"test"}],"waiting_count":12032,"post_count":0,"discussed_count":0,"releases":[{"acquired":"2026-09-11","revision":"b189ec765aa7ed75c8acfea42df31fdae71f97be","release_id":"f17f6fbd935b262bc12f1ea83a040baccef747b90d56b3dcd0b02272c95790ce"},{"acquired":"2026-09-11","revision":"b189ec765aa7ed75c8acfea42df31fdae71f97be","release_id":"518b10e2176c7c288ada93ec5eebcade11408c286b868c04bbe8253e85cec8c0"},{"acquired":"2026-09-11","revision":"b189ec765aa7ed75c8acfea42df31fdae71f97be","release_id":"c2f53d6ad184d253f6db64f70f1a0a1ec554ba4d1ebe1c5153d69ffa1e148cb0"}],"terms":{"license":null,"checked_at":null,"distribution_terms":null,"publisher_requests":null}}},{"benchmark":{"slug":"musr","formal_name":"MuSR","category":"Reasoning","effective_mode":"full","profile":{"resources":[{"kind":"dataset","url":"https://huggingface.co/datasets/TAUR-Lab/MuSR","label":"Source dataset"},{"kind":"issues","url":"https://huggingface.co/datasets/TAUR-Lab/MuSR/discussions","label":"Dataset discussions"},{"kind":"site","url":"https://zayne-sprague.github.io/MuSR/","label":"Official site"},{"kind":"paper","url":"https://arxiv.org/abs/2310.16049","label":"arXiv:2310.16049"}],"task_format":"A narrative of several hundred words, a question about it, and the candidate answers as a list. The correct index is not part of the prompt.","scoring":"The source ships the answer index and the chosen option; this catalogue keeps both in restricted storage and never renders them.","metric":"accuracy over the 250 problems in each domain","size":"756 problems imported across the three domain files (murder mysteries, object placements and team allocation).","answer_access":"Columns outside the prompt allowlist, including any answer or reference solution, are separated into restricted storage and never rendered.","license":"cc-by-4.0 (declared on the Hugging Face dataset card)","history":"Zayne Sprague and collaborators introduced MuSR in 2023 to study multistep reasoning in narratives where individual clues require commonsense interpretation. The benchmark covered murder mysteries, object placement, and team allocation rather than relying only on explicitly stated logical rules. The catalogue lists TAUR-Lab as the source maintainer and records the Hugging Face dataset publication in May 2024.","construction":"MuSR uses algorithmically generated narratives across three domains: murder mysteries, object placement and team allocation. Each problem presents a narrative, a question and candidate answers, while the reference data identifies the correct option. This catalogue records 250 problems in the murder-mystery selection and imports 200 of them, rather than representing all three domains.","limitations":"The generated narratives provide a controlled setting but do not establish performance on unrestricted real-world reasoning. Multiple-choice accuracy measures selection of the reference option without directly verifying the reasoning used to reach it. The murder-mystery selection also cannot substitute for an aggregate across object placement and team allocation, whose reasoning demands differ.","saturation":"","citation":"@article{sprague2023musr,\n  title={MuSR: Testing the Limits of Chain-of-thought with Multistep Soft Reasoning},\n  author={Zayne Sprague and Xi Ye and Kaj Bostrom and Swarat Chaudhuri and Greg Durrett},\n  journal={arXiv preprint arXiv:2310.16049},\n  year={2023}\n}","maintainer":"TAUR-Lab (source organisation)","released":"Dataset published on Hugging Face 2024-05","why_hard":"The clues are spread through a long narrative and none of them decides the answer alone. The chain has to be assembled from commonsense inferences that the text never states.","related":["drop","zebralogicbench","longbench-v2"],"scores":[]}},"facts":{"slug":"musr","task_count":756,"image_count":0,"formats":[{"count":756,"value":"text"}],"answers":[{"count":756,"value":"published"}],"configs":[{"count":756,"value":"default"}],"splits":[{"count":250,"value":"murder_mysteries"},{"count":256,"value":"object_placements"},{"count":250,"value":"team_allocation"}],"waiting_count":756,"post_count":0,"discussed_count":0,"releases":[{"acquired":"2026-09-23","revision":"7c365b439a222150f317764d4f16ae6c96d7d94a","release_id":"4eea9fa15522973c1f478de8183f0045278ab5d6fbf6cbe9f3dca7239bed239c"},{"acquired":"2026-09-12","revision":"7c365b439a222150f317764d4f16ae6c96d7d94a","release_id":"2a1b3a07a10a6004c55a8d6db2226ecfc98e4fcc755e48ba37451484ea54dd53"}],"terms":{"license":null,"checked_at":null,"distribution_terms":null,"publisher_requests":null}}},{"benchmark":{"slug":"winogrande","formal_name":"WinoGrande","category":"Reasoning","effective_mode":"metadata_only","profile":{"resources":[{"kind":"dataset","url":"https://huggingface.co/datasets/allenai/winogrande","label":"Source dataset"},{"kind":"issues","url":"https://huggingface.co/datasets/allenai/winogrande/discussions","label":"Dataset discussions"},{"kind":"repository","url":"https://github.com/allenai/winogrande","label":"Official repository"},{"kind":"paper","url":"https://arxiv.org/abs/1907.10641","label":"arXiv:1907.10641"}],"task_format":"A sentence with a blank and two candidate fillers.","scoring":"The source ships the correct option; nothing is imported here.","metric":"accuracy, reported across training-set sizes from 160 to 40,000 examples","size":"About 44,000 problems in the source; none imported.","answer_access":"No problem text is imported, so there is no answer storage for this entry. The source distributes the answers under its own terms.","license":"The dataset card has a Licensing Information section but names no licence.","history":"The Allen Institute for Artificial Intelligence introduced WinoGrande in 2019 to scale up commonsense language understanding evaluation inspired by the Winograd Schema Challenge. The project sought a large collection with fewer exploitable statistical cues than ordinary crowdsourced data. The March 2022 Hugging Face publication recorded here refers to a later distribution.","construction":"WinoGrande was built from crowdsourced sentence-completion problems and subjected to adversarial filtering to reduce dataset biases. The source contains about 44,000 problems, each presenting a sentence with a blank and two candidate fillers. Evaluation protocols include different amounts of training data, ranging from 160 to 40,000 examples.","limitations":"Adversarial filtering reduces cues detectable by the filtering procedure; it does not establish that all statistical shortcuts have been removed. Binary sentence completion measures a constrained form of commonsense language understanding rather than open-ended reasoning about the world. Scores also depend on the amount of task-specific training data, so those protocols should be distinguished from few-shot prompting.","saturation":"The recorded vendor report gives DeepSeek-V3 base 84.9 percent accuracy with five examples in the prompt. That result leaves headroom under the stated protocol, although the two-choice format has a 50 percent uniform-guessing baseline. A single few-shot report does not establish the best achievable result under the supervised training protocols.","citation":"@article{sakaguchi2019winogrande,\n  title={WinoGrande: An Adversarial Winograd Schema Challenge at Scale},\n  author={Keisuke Sakaguchi and Ronan Le Bras and Chandra Bhagavatula and Yejin Choi},\n  journal={arXiv preprint arXiv:1907.10641},\n  year={2019}\n}","maintainer":"allenai (source organisation)","released":"Dataset published on Hugging Face 2022-03","why_hard":"The two sentences differ by a single word, so every cue except the intended commonsense one is held constant.","related":["hellaswag","arc-challenge","musr"],"scores":[{"model":"DeepSeek-V3","organization":"DeepSeek","score":84.9,"metric":"Acc.","setup":"Base model; 5-shot.","reported_by":"vendor","source_url":"https://raw.githubusercontent.com/deepseek-ai/DeepSeek-V3/main/README.md","source_label":"Base Model, Standard Benchmarks, WinoGrande (Acc.), final column","reported_at":"","checked_at":"2026-09-16"}]}},"facts":null},{"benchmark":{"slug":"zebralogicbench","formal_name":"ZebraLogicBench","category":"Reasoning","effective_mode":"full","profile":{"resources":[{"kind":"dataset","url":"https://huggingface.co/datasets/allenai/ZebraLogicBench","label":"Source dataset"},{"kind":"issues","url":"https://huggingface.co/datasets/allenai/ZebraLogicBench/discussions","label":"Dataset discussions"}],"task_format":"Logic-puzzle text in puzzle with a size field; this selection uses grid_mode for attribute assignments.","scoring":"The published dataset masks the answer, so no answer key exists to check against here.","metric":"","size":"1,000 grid-mode puzzles imported from the grid_mode / test selection (the full split).","answer_access":"The solution table, when supplied, is kept outside the puzzle prompt in restricted storage. Empty or masked solution rows do not establish answer availability.","license":"","history":"The allenai organisation published the ZebraLogicBench dataset on Hugging Face in July 2024. It evaluates reasoning through logic puzzles in which clues constrain the assignment of attributes to entries in a grid. The task focuses on combining constraints consistently rather than recalling an isolated fact.","construction":"","limitations":"The grid-mode task measures a particular form of constraint reasoning rather than general factual knowledge or open-ended problem solving. The source card distinguishes grid mode from a separate multiple-choice setting, so results need to identify which format was used. For the selection described in this catalogue, published answers are masked; those records alone therefore do not provide an answer key for independently verifying accuracy.","saturation":"","citation":"","maintainer":"allenai (Hugging Face organisation)","released":"Dataset published on Hugging Face 2024-07","why_hard":"Requires combining clue constraints to determine attribute assignments.","related":["hle","arc-agi-2","bbeh"],"scores":[]}},"facts":{"slug":"zebralogicbench","task_count":1000,"image_count":0,"formats":[{"count":1000,"value":"text"}],"answers":[{"count":1000,"value":"unavailable"}],"configs":[{"count":1000,"value":"grid_mode"}],"splits":[{"count":1000,"value":"test"}],"waiting_count":1000,"post_count":0,"discussed_count":0,"releases":[{"acquired":"2026-09-09","revision":"2f94a445d7079f20146f5443e2606049de8543e0","release_id":"77772c4c40c8d8d6e6aca8cccab2a7f933c44ab8cdce1c1948c865c4587e0215"},{"acquired":"2026-09-09","revision":"2f94a445d7079f20146f5443e2606049de8543e0","release_id":"7753e95d678b0a62098e8097ae1feceb281a8d37ab0854696ca48bd1167ae2a7"},{"acquired":"2026-09-09","revision":"2f94a445d7079f20146f5443e2606049de8543e0","release_id":"4dd8c6dc1d41af4f8f36da13c74aead6ecd017cad7dd87bed39707527308d7ea"},{"acquired":"2026-09-09","revision":"2f94a445d7079f20146f5443e2606049de8543e0","release_id":"db5c2502d242d4e064a5fa4f5da227cfccdbc94214efab11c8992686bd2c55ff"}],"terms":{"license":null,"checked_at":null,"distribution_terms":null,"publisher_requests":null}}},{"benchmark":{"slug":"arc-agi-3","formal_name":"ARC-AGI-3","category":"Reasoning / Agents","effective_mode":"metadata_only","profile":{"resources":[{"kind":"site","url":"https://arcprize.org/arc-agi/3/","label":"Official site"}],"task_format":"Interactive game environments accessed through an official toolkit, with game versions and trial records tracked separately.","scoring":"","metric":"","size":"","answer_access":"","license":"","history":"","construction":"","limitations":"Performance depends on interaction with a game environment, so the score combines rule discovery, exploration and action planning rather than isolating any one of them. Comparisons need to account for game versions and evaluation conditions, including which tools are supplied. The recorded release results use a semi-private set and a shared system prompt with no tools supplied by the evaluator; they do not establish performance under other agent configurations.","saturation":"The recorded official release leaderboard reports scores of 0.37% for Gemini 3.1 Pro Preview, 0.26% for GPT 5.4 (High) and 0.25% for Opus 4.6 (Max) on the semi-private set. These reported scores are far from the top of the percentage scale and provide no indication of saturation in that setup. They are benchmark scores, not evidence that the same percentages of complete games were solved.","citation":"","maintainer":"","released":"","why_hard":"Requires exploring unfamiliar game environments, learning rules, and planning actions.","related":["hle","arc-agi-2","bbeh"],"scores":[{"model":"Gemini 3.1 Pro Preview","organization":"Google","score":0.37,"metric":"Score (%)","setup":"Semi-private set; official leaderboard at release; same system prompt, no tools supplied by the evaluator.","reported_by":"paper","source_url":"https://arxiv.org/html/2603.24621v1","source_label":"Table 2; Section 4.3.1","reported_at":"","checked_at":"2026-09-16"},{"model":"GPT 5.4 (High)","organization":"OpenAI","score":0.26,"metric":"Score (%)","setup":"Semi-private set; official leaderboard at release; same system prompt, no tools supplied by the evaluator.","reported_by":"paper","source_url":"https://arxiv.org/html/2603.24621v1","source_label":"Table 2; Section 4.3.1","reported_at":"","checked_at":"2026-09-16"},{"model":"Opus 4.6 (Max)","organization":"Anthropic","score":0.25,"metric":"Score (%)","setup":"Semi-private set; official leaderboard at release; same system prompt, no tools supplied by the evaluator.","reported_by":"paper","source_url":"https://arxiv.org/html/2603.24621v1","source_label":"Table 2; Section 4.3.1","reported_at":"","checked_at":"2026-09-16"}]}},"facts":null},{"benchmark":{"slug":"ifeval","formal_name":"IFEval","category":"Reasoning / Instruction following","effective_mode":"full","profile":{"resources":[{"kind":"dataset","url":"https://huggingface.co/datasets/google/IFEval","label":"Source dataset"},{"kind":"issues","url":"https://huggingface.co/datasets/google/IFEval/discussions","label":"Dataset discussions"},{"kind":"paper","url":"https://arxiv.org/abs/2311.07911","label":"arXiv:2311.07911"}],"task_format":"A natural-language prompt carrying one or more machine-checkable constraints, with the constraint identifiers recorded alongside it.","scoring":"Every column outside the prompt is separated into restricted storage, but the converter does not interpret it, so this catalogue does not claim a usable answer key exists.","metric":"","size":"541 problems imported from the default / train selection (the full split; the source names its only split train).","answer_access":"Columns outside the prompt allowlist, including any answer or reference solution, are separated into restricted storage and never rendered.","license":"apache-2.0 (declared on the source)","history":"Google researchers introduced IFEval in 2023 to measure instruction following with constraints that can be checked automatically. The motivation was to make compliance with explicit instructions measurable without relying entirely on subjective judgments of answer quality. The Google dataset was published on Hugging Face in December 2023.","construction":"IFEval prompts contain one or more verifiable constraints, such as requirements about word counts, output format or the presence of particular words. Constraint identifiers and associated checking information accompany the natural-language prompts so that compliance can be assessed programmatically. This catalog imports 200 prompts from the source selection labeled train; that import count is not the size of the complete benchmark.","limitations":"Passing formal constraints does not establish that an answer is accurate, relevant or useful. Conversely, an otherwise good answer can fail because it violates a requested format. Reported results must distinguish instruction-level from prompt-level scoring, since satisfying some constraints is different from satisfying every constraint in a prompt. Strict and loose checking variants can also produce different scores and should not be treated as interchangeable.","saturation":"","citation":"@misc{zhou2023instructionfollowingevaluationlargelanguage,\n      title={Instruction-Following Evaluation for Large Language Models}, \n      author={Jeffrey Zhou and Tianjian Lu and Swaroop Mishra and Siddhartha Brahma and Sujoy Basu and Yi Luan and Denny Zhou and Le Hou},\n      year={2023},\n      eprint={2311.07911},\n      archivePrefix={arXiv},\n      primaryClass={cs.CL},\n      url={https://arxiv.org/abs/2311.07911}, \n}","maintainer":"google (source organisation)","released":"Dataset published on Hugging Face 2023-12","why_hard":"The constraints are easy to state and easy to violate in passing; models routinely answer well while breaking the format they were told to use.","related":["arc-agi-2","arc-agi-3","bbeh"],"scores":[]}},"facts":{"slug":"ifeval","task_count":541,"image_count":0,"formats":[{"count":541,"value":"text"}],"answers":[{"count":541,"value":"unknown"}],"configs":[{"count":541,"value":"default"}],"splits":[{"count":541,"value":"train"}],"waiting_count":541,"post_count":0,"discussed_count":0,"releases":[{"acquired":"2026-09-11","revision":"966cd89545d6b6acfd7638bc708b98261ca58e84","release_id":"54f3548aa995f46fa9e9cb5f8d6b498230cb6a14666bcbdfe034cfe22b129e7e"},{"acquired":"2026-09-11","revision":"966cd89545d6b6acfd7638bc708b98261ca58e84","release_id":"a7368be0a38aad1946db5c4085b7bcf7b47239eade1e1de4805fc13e366c924d"},{"acquired":"2026-09-11","revision":"966cd89545d6b6acfd7638bc708b98261ca58e84","release_id":"dd89911e924b0eaba97db2a0acf5662e164f59d47fd0c4d7148d3d1ec27e553f"},{"acquired":"2026-09-11","revision":"966cd89545d6b6acfd7638bc708b98261ca58e84","release_id":"626f175959b672835a1eacfe5080de1ce89ace42f41675f2bc9ecfaa43b2c2d4"}],"terms":{"license":null,"checked_at":null,"distribution_terms":null,"publisher_requests":null}}},{"benchmark":{"slug":"mmlu","formal_name":"MMLU","category":"Reasoning / Knowledge","effective_mode":"full","profile":{"resources":[{"kind":"dataset","url":"https://huggingface.co/datasets/cais/mmlu","label":"Source dataset"},{"kind":"issues","url":"https://huggingface.co/datasets/cais/mmlu/discussions","label":"Dataset discussions"},{"kind":"repository","url":"https://github.com/hendrycks/test","label":"Official repository"},{"kind":"paper","url":"https://arxiv.org/abs/2009.03300","label":"arXiv:2009.03300"}],"task_format":"A question with four options, plus the subject it belongs to. The options are imported as a list; the index of the correct one is not.","scoring":"The source ships the answer index with every problem; this catalogue keeps it in restricted storage and never renders it.","metric":"accuracy (fraction of questions answered correctly, usually reported as a macro average over the 57 subjects)","size":"14,015 unique problems imported from the all / test selection of 14,042 rows; 27 source rows repeat an earlier question exactly and are recorded as variants. The 57 subject configurations overlap this selection and are not imported separately.","answer_access":"Columns outside the prompt allowlist, including any answer or reference solution, are separated into restricted storage and never rendered.","license":"mit (declared on the Hugging Face dataset card)","history":"Dan Hendrycks and collaborators introduced MMLU in 2020 to test how broadly language models could apply knowledge across academic and professional subjects. The benchmark spans 57 subjects, from elementary material to professional examinations. Its motivation was to evaluate breadth and depth of learned knowledge beyond the narrower tasks common in earlier language-understanding benchmarks.","construction":"MMLU collects four-option multiple-choice questions from existing educational and examination materials across 57 subjects. Each item has a subject label and a reference answer index. The source test selection contains 14,042 questions, while this catalog imports only 200 of them. Accuracy measures correct answer selection, with results commonly summarized across subjects.","limitations":"The multiple-choice format permits guessing and does not require a model to justify its answer or perform the corresponding professional task. Publicly available source materials and benchmark questions create a risk of overlap with training data, although overlap alone does not establish that a particular model memorized the test. MMLU has also attracted criticism over ambiguous questions and incorrect reference answers. Prompting, answer extraction and subject aggregation can affect comparisons between reports.","saturation":"The recorded vendor result for the DeepSeek-V3 base model reports 87.1% accuracy in a five-shot setup. This is a high score but still leaves errors on a substantial minority of questions, and one entry is insufficient to establish the benchmark's current ceiling. As results rise, remaining label ambiguities and differences in evaluation procedure become more consequential when interpreting small score gaps.","citation":"@article{hendrycks2020mmlu,\n  title={Measuring Massive Multitask Language Understanding},\n  author={Dan Hendrycks and Collin Burns and Steven Basart and Andy Zou and Mantas Mazeika and Dawn Song and Jacob Steinhardt},\n  journal={arXiv preprint arXiv:2009.03300},\n  year={2020}\n}","maintainer":"cais (source organisation)","released":"Dataset published on Hugging Face 2022-03","why_hard":"No single subject is hard; the breadth is. A model must hold professional-level material in law, medicine and mathematics at once, and the four-way format leaves little room to recover from a gap in knowledge.","related":["mmlu-pro","arc-challenge","hle"],"scores":[{"model":"DeepSeek-V3","organization":"DeepSeek","score":87.1,"metric":"Acc.","setup":"Base model; 5-shot.","reported_by":"vendor","source_url":"https://raw.githubusercontent.com/deepseek-ai/DeepSeek-V3/main/README.md","source_label":"Base Model, Standard Benchmarks, MMLU (Acc.), final column","reported_at":"","checked_at":"2026-09-16"}]}},"facts":{"slug":"mmlu","task_count":14015,"image_count":0,"formats":[{"count":14015,"value":"text"}],"answers":[{"count":14015,"value":"published"}],"configs":[{"count":14015,"value":"all"}],"splits":[{"count":14015,"value":"test"}],"waiting_count":14015,"post_count":0,"discussed_count":0,"releases":[{"acquired":"2026-09-12","revision":"c30699e8356da336a370243923dbaf21066bb9fe","release_id":"81fc4de3e8e3b3c1faf438f4d0bd3e0a253144d760982f7c251eb9ab9afb4349"},{"acquired":"2026-09-12","revision":"c30699e8356da336a370243923dbaf21066bb9fe","release_id":"8cfe6e6995d0a76b4a50dfb1d8e0125e2bb38d35b2a56a47b7959009d8030e11"}],"terms":{"license":null,"checked_at":null,"distribution_terms":null,"publisher_requests":null}}},{"benchmark":{"slug":"longbench-v2","formal_name":"LongBench v2","category":"Reasoning / Long context","effective_mode":"full","profile":{"resources":[{"kind":"dataset","url":"https://huggingface.co/datasets/zai-org/LongBench-v2","label":"Source dataset"},{"kind":"issues","url":"https://huggingface.co/datasets/zai-org/LongBench-v2/discussions","label":"Dataset discussions"},{"kind":"paper","url":"https://arxiv.org/abs/2412.15204","label":"arXiv:2412.15204"}],"task_format":"Multiple-choice questions with context, choice_A through choice_D, and optional domain, sub_domain, difficulty, and length fields.","scoring":"For the 200 imported problems, the source ships an answer with every problem; this catalogue keeps it in restricted storage and never renders it. Automated evaluation is provided by the LongBench repository at https://github.com/THUDM/LongBench.","metric":"accuracy over multiple-choice questions (README: 503 multiple-choice questions)","size":"503 questions described in the catalog introduction; this train selection expects 200 task records with a source-row limit of 200.","answer_access":"Fields outside the prompt allowlist, including answers when supplied, remain in restricted storage; the converter alone does not establish whether usable answers are available.","license":"apache-2.0 (declared on the Hugging Face dataset card)","history":"LongBench v2 was introduced in 2024 as a benchmark for deep understanding and reasoning over long contexts. The dataset distributed by zai-org on Hugging Face was published in December 2024. It addresses tasks such as single-document and multi-document question answering and code-repository understanding, extending evaluation beyond merely locating a fact in a long input.","construction":"","limitations":"The four-option format measures answer selection rather than the ability to produce a supported explanation or a complete working change to a code repository. A correct selection does not reveal which parts of the context supported it. Results also depend on how an evaluation handles contexts that exceed a model's input limit. The official collection contains 503 questions; this catalog's 200 imported records are a limited selection and should not be confused with a full-benchmark evaluation.","saturation":"","citation":"@article{bai2024longbench2,\n  title={LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks}, \n  author={Yushi Bai and Shangqing Tu and Jiajie Zhang and Hao Peng and Xiaozhi Wang and Xin Lv and Shulin Cao and Jiazheng Xu and Lei Hou and Yuxiao Dong and Jie Tang and Juanzi Li},\n  journal={arXiv preprint arXiv:2412.15204},\n  year={2024}\n}","maintainer":"zai-org (Hugging Face organisation)","released":"Dataset published on Hugging Face 2024-12","why_hard":"Requires deep understanding and reasoning over long contexts, including single-document and multi-document QA and code-repository understanding.","related":["hle","arc-agi-2","bbeh"],"scores":[]}},"facts":{"slug":"longbench-v2","task_count":200,"image_count":0,"formats":[{"count":200,"value":"text"}],"answers":[{"count":200,"value":"published"}],"configs":[{"count":200,"value":""}],"splits":[{"count":200,"value":"train"}],"waiting_count":200,"post_count":0,"discussed_count":0,"releases":[{"acquired":"2026-09-09","revision":"2b48e494f2c7a2f0af81aae178e05c7e1dde0fe9","release_id":"163e4eea78f97c552299481ff2990b359219ce6513c67e0994d8a69c09b95a76"},{"acquired":"2026-09-09","revision":"2b48e494f2c7a2f0af81aae178e05c7e1dde0fe9","release_id":"2154326d7cb82e6db3711e4e20565c8fe5237a49f6d22099434252fea7c31bb4"},{"acquired":"2026-09-09","revision":"2b48e494f2c7a2f0af81aae178e05c7e1dde0fe9","release_id":"052dc8a10560780f5d1ae66a2e914012ca5e5ed601996886701ca2302237223f"},{"acquired":"2026-09-09","revision":"2b48e494f2c7a2f0af81aae178e05c7e1dde0fe9","release_id":"0a6d2989e8ed03663bd7b3efc800dc97e4fa7785267e4e7b318bbf1659afc0bc"}],"terms":{"license":null,"checked_at":null,"distribution_terms":null,"publisher_requests":null}}},{"benchmark":{"slug":"ruler","formal_name":"RULER","category":"Reasoning / Long context","effective_mode":"metadata_only","profile":{"resources":[{"kind":"repository","url":"https://github.com/NVIDIA/RULER","label":"Official repository"},{"kind":"paper","url":"https://arxiv.org/abs/2404.06654","label":"arXiv:2404.06654"}],"task_format":"Synthetic long documents with planted information to retrieve, trace, or aggregate.","scoring":"Generated with its own answer key; nothing is imported here.","metric":"accuracy per task at each context length, and the effective length at which performance falls below a threshold","size":"Generated per run; nothing fixed to import.","answer_access":"No problem text is imported, so there is no answer storage for this entry. The source distributes the answers under its own terms.","license":"apache-2.0 (declared by the LICENSE file in the official NVIDIA/RULER repository)","history":"NVIDIA researchers introduced RULER in 2024 to investigate how much of an advertised context window a language model could actually use. The benchmark broadened long-context testing beyond simple retrieval to include tracing, aggregation and question answering. Its purpose was to expose task-dependent failures that a nominal maximum input length does not describe.","construction":"RULER generates synthetic long documents containing information that must be retrieved, traced or aggregated to answer a task. Generation provides the corresponding answer key and allows evaluations at different context lengths instead of relying on one fixed document collection. The benchmark reports accuracy by task and length, with effective context length defined through a performance threshold.","limitations":"Synthetic contexts isolate particular information-processing demands but do not reproduce the full complexity of long documents or sustained real-world workflows. An effective-length result depends on the selected tasks, generation settings and threshold, rather than being an intrinsic constant of a model. Strong performance on a simple retrieval task therefore does not establish equivalent ability to combine or trace information over the same context length.","saturation":"","citation":"@article{hsieh2024ruler,\n  title={RULER: What's the Real Context Size of Your Long-Context Language Models?},\n  author={Cheng-Ping Hsieh and Simeng Sun and Samuel Kriman and Shantanu Acharya and Dima Rekesh and Fei Jia and Yang Zhang and Boris Ginsburg},\n  journal={arXiv preprint arXiv:2404.06654},\n  year={2024}\n}","maintainer":"NVIDIA (source organisation)","released":"","why_hard":"It separates the advertised context window from the usable one. Most models degrade well before the limit they claim, and the point of failure depends on the task, not just the length.","related":["longbench-v2","hle","arc-agi-2"],"scores":[]}},"facts":null},{"benchmark":{"slug":"hle","formal_name":"Humanity's Last Exam","category":"Reasoning / Multimodal","effective_mode":"metadata_only","profile":{"resources":[{"kind":"dataset","url":"https://huggingface.co/datasets/cais/hle","label":"Source dataset"},{"kind":"issues","url":"https://huggingface.co/datasets/cais/hle/discussions","label":"Dataset discussions"},{"kind":"site","url":"https://lastexam.ai/","label":"Official site"}],"task_format":"Text and image questions across academic disciplines.","scoring":"","metric":"","size":"","answer_access":"This catalog has not downloaded questions or answers; its acquisition record notes gated access and a publisher request against republication.","license":"mit (declared on the Hugging Face dataset card)","history":"The Center for AI Safety and Scale AI introduced Humanity's Last Exam in 2025 to provide a difficult test of academic knowledge and reasoning as existing benchmarks became less discriminating. The dataset was published on Hugging Face in January 2025. Its questions span academic disciplines and include both text-only and image-based material.","construction":"Humanity's Last Exam was assembled from questions contributed by subject-matter experts across academic disciplines. Candidate questions underwent expert review and screening against contemporary models to select challenging items. The resulting evaluation includes text and image questions, so success can require both specialist knowledge and interpretation of visual material.","limitations":"Performance on difficult academic questions does not directly measure sustained research, experimental design or successful action in an external environment. Screening questions for difficulty against existing models also makes the collection a deliberately selected challenge set rather than a representative sample of everyday knowledge tasks. The recorded leaderboard uses an updated dataset and an o3-mini judge, so dataset revision and answer-grading procedure are part of the evaluation conditions.","saturation":"The recorded official leaderboard reports 38.3% accuracy for Gemini 3 Pro, 25.3% for GPT-5 and 24.5% for Grok 4. These entries use the dataset updated on April 3, 2025 and an o3-mini judge. The reported results leave substantial headroom under that setup, while comparisons with other reports require matching the dataset version and grading conditions.","citation":"","maintainer":"cais (Hugging Face organisation)","released":"Dataset published on Hugging Face 2025-01","why_hard":"Requires knowledge and reasoning across academic disciplines.","related":["arc-agi-2","bbeh","gaia"],"scores":[{"model":"Gemini 3 Pro","organization":"","score":38.3,"metric":"Accuracy (%)","setup":"Dataset updated April 3, 2025; judge model o3-mini.","reported_by":"leaderboard","source_url":"https://lastexam.ai/","source_label":"Quantitative Results, Accuracy column","reported_at":"","checked_at":"2026-09-16"},{"model":"GPT-5","organization":"","score":25.3,"metric":"Accuracy (%)","setup":"Dataset updated April 3, 2025; judge model o3-mini.","reported_by":"leaderboard","source_url":"https://lastexam.ai/","source_label":"Quantitative Results, Accuracy column","reported_at":"","checked_at":"2026-09-16"},{"model":"Grok 4","organization":"","score":24.5,"metric":"Accuracy (%)","setup":"Dataset updated April 3, 2025; judge model o3-mini.","reported_by":"leaderboard","source_url":"https://lastexam.ai/","source_label":"Quantitative Results, Accuracy column","reported_at":"","checked_at":"2026-09-16"}]}},"facts":null},{"benchmark":{"slug":"truthfulqa","formal_name":"TruthfulQA","category":"Reasoning / Safety","effective_mode":"full","profile":{"resources":[{"kind":"dataset","url":"https://huggingface.co/datasets/truthfulqa/truthful_qa","label":"Source dataset"},{"kind":"issues","url":"https://huggingface.co/datasets/truthfulqa/truthful_qa/discussions","label":"Dataset discussions"},{"kind":"repository","url":"https://github.com/sylinrl/TruthfulQA","label":"Official repository"},{"kind":"paper","url":"https://arxiv.org/abs/2109.07958","label":"arXiv:2109.07958"}],"task_format":"One question, written to invite a plausible falsehood. Its category, its adversarial type and the source documenting the misconception all narrow the answer, so none of them is published.","scoring":"The source ships a best answer plus lists of correct and incorrect answers; this catalogue keeps all of them in restricted storage and never renders them.","metric":"truthfulness and informativeness, judged per answer; the multiple-choice configuration reports MC1/MC2 accuracy","size":"817 problems imported from the generation / validation selection (the full validation split). The multiple-choice configuration covers the same questions and is not imported separately.","answer_access":"Columns outside the prompt allowlist, including any answer or reference solution, are separated into restricted storage and never rendered.","license":"apache-2.0 (declared on the Hugging Face dataset card)","history":"Stephanie Lin, Jacob Hilton, and Owain Evans introduced TruthfulQA in 2021 to investigate whether language models reproduce common human falsehoods. The benchmark targeted the possibility that learning to imitate human text also teaches a model to repeat misconceptions. Its research question concerned truthfulness rather than merely producing plausible or fluent answers.","construction":"The authors wrote questions intended to invite plausible false answers associated with common misconceptions. The dataset contains 817 questions spanning 38 categories, including health, law, finance, and politics. Reference material includes a best answer and lists of correct and incorrect answers. This catalogue selects 200 records from the generation configuration and keeps those references outside the prompt.","limitations":"TruthfulQA is deliberately focused on misconception-eliciting questions, so its score does not estimate the frequency of false statements in ordinary use. Generation evaluations distinguish truthfulness from informativeness because an uninformative answer can avoid making a false claim. The multiple-choice configurations and free-form generation assess different behavior and should not be treated as interchangeable metrics. Public questions and reference answers also make training exposure a concern when interpreting later results.","saturation":"","citation":"@article{lin2021truthfulqa,\n  title={TruthfulQA: Measuring How Models Mimic Human Falsehoods},\n  author={Stephanie Lin and Jacob Hilton and Owain Evans},\n  journal={arXiv preprint arXiv:2109.07958},\n  year={2021}\n}","maintainer":"truthfulqa (source organisation)","released":"Dataset published on Hugging Face 2022-06","why_hard":"Every question has an answer that sounds right because people repeat it. Scale alone tends to make this worse, because the falsehood is well represented in the training text.","related":["hle","jailbreakbench","arc-agi-2"],"scores":[]}},"facts":{"slug":"truthfulqa","task_count":817,"image_count":0,"formats":[{"count":817,"value":"text"}],"answers":[{"count":817,"value":"published"}],"configs":[{"count":817,"value":"generation"}],"splits":[{"count":817,"value":"validation"}],"waiting_count":817,"post_count":0,"discussed_count":0,"releases":[{"acquired":"2026-09-12","revision":"741b8276f2d1982aa3d5b832d3ee81ed3b896490","release_id":"7e98884f0664666accde434ce90f7caf45e5280f23a4c9b2c564733a87c8b3d8"},{"acquired":"2026-09-12","revision":"741b8276f2d1982aa3d5b832d3ee81ed3b896490","release_id":"28ecf922883b61babb5d63439ddf85b74ab1416d0797bcd084873acb8c8feac7"}],"terms":{"license":null,"checked_at":null,"distribution_terms":null,"publisher_requests":null}}},{"benchmark":{"slug":"arc-challenge","formal_name":"ARC-Challenge","category":"Reasoning / Science","effective_mode":"full","profile":{"resources":[{"kind":"dataset","url":"https://huggingface.co/datasets/allenai/ai2_arc","label":"Source dataset"},{"kind":"issues","url":"https://huggingface.co/datasets/allenai/ai2_arc/discussions","label":"Dataset discussions"},{"kind":"paper","url":"https://arxiv.org/abs/1803.05457","label":"arXiv:1803.05457"}],"task_format":"A science question with three to five options. The options are imported as a list; the answer key is not.","scoring":"The source ships an answer key with every problem; this catalogue keeps it in restricted storage and never renders it.","metric":"","size":"1,172 problems imported from the ARC-Challenge / test selection (the full test split).","answer_access":"Columns outside the prompt allowlist, including any answer or reference solution, are separated into restricted storage and never rendered.","license":"cc-by-sa-4.0 (declared on the Hugging Face dataset card)","history":"The Allen Institute for Artificial Intelligence introduced the AI2 Reasoning Challenge in 2018 to test question answering that required more than straightforward retrieval. ARC-Challenge is its difficult subset of grade-school science questions. The March 2022 date recorded for the Hugging Face distribution is an upload date rather than the original introduction.","construction":"ARC collects multiple-choice questions from existing science examinations. The Challenge subset retains questions that both a retrieval baseline and a word co-occurrence baseline answered incorrectly. Questions have three to five answer options, and the source test split contains 1,172 items; the catalogue selection contains 200 of these.","limitations":"Failure by the original baselines is a selection criterion, not proof that every retained question requires multi-step reasoning. Multiple-choice accuracy combines scientific knowledge, reasoning and exploitation of answer cues, without directly assessing explanations. Public availability creates a risk of training exposure, and prompting or option-scoring conventions can affect comparisons.","saturation":"The recorded vendor report gives DeepSeek-V3 base 95.3 percent accuracy with 25 examples in the prompt. That result leaves little headroom in that particular setup, limiting the room for aggregate accuracy to distinguish models. It does not establish equivalent performance under other prompting protocols or on science questions outside the fixed test set.","citation":"@article{clark2018arcchallenge,\n  title={Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge},\n  author={Peter Clark and Isaac Cowhey and Oren Etzioni and Tushar Khot and Ashish Sabharwal and Carissa Schoenick and Oyvind Tafjord},\n  journal={arXiv preprint arXiv:1803.05457},\n  year={2018}\n}","maintainer":"allenai (source organisation)","released":"Dataset published on Hugging Face 2022-03","why_hard":"The set was built by discarding everything the simple baselines could already do, so what remains needs a chain of reasoning over the science involved rather than a lookup.","related":["mmlu","gpqa","hellaswag"],"scores":[{"model":"DeepSeek-V3","organization":"DeepSeek","score":95.3,"metric":"Acc.","setup":"Base model; 25-shot.","reported_by":"vendor","source_url":"https://raw.githubusercontent.com/deepseek-ai/DeepSeek-V3/main/README.md","source_label":"Base Model, Standard Benchmarks, ARC-Challenge (Acc.), final column","reported_at":"","checked_at":"2026-09-16"}]}},"facts":{"slug":"arc-challenge","task_count":1172,"image_count":0,"formats":[{"count":1172,"value":"text"}],"answers":[{"count":1172,"value":"published"}],"configs":[{"count":1172,"value":"ARC-Challenge"}],"splits":[{"count":1172,"value":"test"}],"waiting_count":1172,"post_count":0,"discussed_count":0,"releases":[{"acquired":"2026-09-12","revision":"210d026faf9955653af8916fad021475a3f00453","release_id":"64068a43aa5737d5c4c43311e0b54396bb98e773d63c0e66b90d86975a0af472"},{"acquired":"2026-09-12","revision":"210d026faf9955653af8916fad021475a3f00453","release_id":"78d02a92bc4002f8bc618f647aa25f87fc889342dc61bb6417983f2375fc5e2e"},{"acquired":"2026-09-12","revision":"210d026faf9955653af8916fad021475a3f00453","release_id":"a14a6faab45b226a540df0f495ca0e1af4175f4d30910684f432c2f2614898b4"}],"terms":{"license":null,"checked_at":null,"distribution_terms":null,"publisher_requests":null}}},{"benchmark":{"slug":"arc-easy","formal_name":"ARC-Easy","category":"Reasoning / Science","effective_mode":"full","profile":{"resources":[{"kind":"dataset","url":"https://huggingface.co/datasets/allenai/ai2_arc","label":"Source dataset"},{"kind":"issues","url":"https://huggingface.co/datasets/allenai/ai2_arc/discussions","label":"Dataset discussions"},{"kind":"paper","url":"https://arxiv.org/abs/1803.05457","label":"arXiv:1803.05457"}],"task_format":"A science question with three to five options, from the subset the simple baselines answered correctly. The options are imported as a list; the answer key is not.","scoring":"The source ships an answer key with every problem; this catalogue keeps it in restricted storage and never renders it.","metric":"","size":"2,376 problems imported from the ARC-Easy / test selection (the full test split).","answer_access":"Columns outside the prompt allowlist, including any answer or reference solution, are separated into restricted storage and never rendered.","license":"cc-by-sa-4.0 (declared on the Hugging Face dataset card)","history":"","construction":"","limitations":"","saturation":"","citation":"@article{clark2018arcchallenge,\n  title={Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge},\n  author={Peter Clark and Isaac Cowhey and Oren Etzioni and Tushar Khot and Ashish Sabharwal and Carissa Schoenick and Oyvind Tafjord},\n  journal={arXiv preprint arXiv:1803.05457},\n  year={2018}\n}","maintainer":"allenai (source organisation)","released":"Dataset published on Hugging Face 2022-03","why_hard":"Being the complement of the Challenge set means the questions reward direct knowledge recall; a model that still misses them has a gap in basic science facts rather than in multi-step reasoning.","related":["arc-challenge","mmlu","hellaswag"],"scores":[]}},"facts":{"slug":"arc-easy","task_count":2376,"image_count":0,"formats":[{"count":2376,"value":"text"}],"answers":[{"count":2376,"value":"published"}],"configs":[{"count":2376,"value":"ARC-Easy"}],"splits":[{"count":2376,"value":"test"}],"waiting_count":2376,"post_count":0,"discussed_count":0,"releases":[{"acquired":"2026-09-23","revision":"210d026faf9955653af8916fad021475a3f00453","release_id":"e20f8b35e6fd59e66e4f904911eafd8473c1567524ef9ef679040bb5d96fe782"}],"terms":{"license":null,"checked_at":null,"distribution_terms":null,"publisher_requests":null}}},{"benchmark":{"slug":"harmbench","formal_name":"HarmBench","category":"Safety","effective_mode":"metadata_only","profile":{"resources":[{"kind":"dataset","url":"https://huggingface.co/datasets/walledai/HarmBench","label":"Source dataset"},{"kind":"issues","url":"https://huggingface.co/datasets/walledai/HarmBench/discussions","label":"Dataset discussions"},{"kind":"paper","url":"https://arxiv.org/abs/2402.04249","label":"arXiv:2402.04249"}],"task_format":"Harmful behaviour descriptions. The mirror linked here carries the 400 textual behaviours in the standard, contextual and copyright categories; the full benchmark adds 110 multimodal behaviours that the mirror does not.","scoring":"Scored by a trained classifier deciding whether a completion actually carries out the behaviour; nothing is imported here.","metric":"attack success rate (ASR) per attack and defence pair","size":"510 behaviours in the full benchmark, of which the linked mirror carries 400 textual ones; none imported.","answer_access":"No problem text is imported, so there is no answer storage for this entry. The source distributes the answers under its own terms.","license":"mit (declared on the Hugging Face dataset card)","history":"Researchers at the Center for AI Safety and collaborators introduced HarmBench in 2024 as a standardized framework for automated red teaming and robust refusal. The aim was to compare attack methods and defenses under a shared evaluation procedure. The walledai dataset linked here is a mirror of textual behaviors, rather than the complete original benchmark.","construction":"The full benchmark contains 510 behaviors, while the linked mirror carries 400 textual behaviors in the standard, contextual and copyright categories. The remaining 110 behaviors are multimodal and are not included in that mirror. Evaluation pairs an attack with a defense and uses a trained classifier to decide whether the resulting completion carries out the specified behavior.","limitations":"Attack success rate is a property of an attack and defense pair under an evaluation procedure, rather than a universal safety score for a model. A trained classifier provides an operational judgment of compliance and can make errors; its verdict does not establish real-world harm or harmlessness. The textual mirror also omits the multimodal portion, so an evaluation on that mirror cannot support conclusions about all 510 behaviors.","saturation":"","citation":"@article{mazeika2024harmbench,\n  title={HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal},\n  author={Mantas Mazeika and Long Phan and Xuwang Yin and Andy Zou and Zifan Wang and Norman Mu and Elham Sakhaee and Nathaniel Li and Steven Basart and Bo Li and David Forsyth and Dan Hendrycks},\n  journal={arXiv preprint arXiv:2402.04249},\n  year={2024}\n}","maintainer":"walledai (source organisation)","released":"Dataset published on Hugging Face 2024-07","why_hard":"It separates a model that refuses from a model that merely sounds like it refused: the classifier judges the completion, so a compliant answer wrapped in a disclaimer still counts as a success for the attack.","related":["jailbreakbench","truthfulqa","agentharm"],"scores":[]}},"facts":null},{"benchmark":{"slug":"jailbreakbench","formal_name":"JailbreakBench","category":"Safety","effective_mode":"metadata_only","profile":{"resources":[{"kind":"dataset","url":"https://huggingface.co/datasets/JailbreakBench/JBB-Behaviors","label":"Source dataset"},{"kind":"issues","url":"https://huggingface.co/datasets/JailbreakBench/JBB-Behaviors/discussions","label":"Dataset discussions"},{"kind":"site","url":"https://jailbreakbench.github.io/","label":"Official site"},{"kind":"paper","url":"https://arxiv.org/abs/2404.01318","label":"arXiv:2404.01318"}],"task_format":"Paired harmful and benign behaviour descriptions, used as targets for attack and defence artefacts.","scoring":"Scored by a judge model against the defined behaviours; nothing is imported here.","metric":"attack success rate, with the benign pairs used to measure over-refusal","size":"100 harmful and 100 benign behaviours in the source; none imported.","answer_access":"No problem text is imported, so there is no answer storage for this entry. The source distributes the answers under its own terms.","license":"mit (declared on the Hugging Face dataset card)","history":"The JailbreakBench project introduced an open benchmark in 2024 for evaluating attacks on language-model safeguards and the defences against them. Its behaviour dataset was published on Hugging Face in June 2024. The evaluation includes benign requests as well as harmful targets so that robustness can be considered alongside unnecessary refusals.","construction":"The source defines 100 harmful behaviours and 100 matched benign behaviours, which serve as targets for attack and defence evaluation. A judge model assesses responses against the defined behaviours. Attack success rate measures harmful compliance, while the benign behaviours support a separate assessment of over-refusal.","limitations":"A finite collection of behaviours cannot establish safety across all possible requests or attack strategies. Attack success rate depends on the attack method, its budget, the target model configuration and the judge used to classify responses. Judge errors can affect the reported rate, and low attack success alone does not establish a useful defence without considering performance on benign requests.","saturation":"","citation":"@article{chao2024jailbreakbench,\n  title={JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models},\n  author={Patrick Chao and Edoardo Debenedetti and Alexander Robey and Maksym Andriushchenko and Francesco Croce and Vikash Sehwag and Edgar Dobriban and Nicolas Flammarion and George J. Pappas and Florian Tramer and Hamed Hassani and Eric Wong},\n  journal={arXiv preprint arXiv:2404.01318},\n  year={2024}\n}","maintainer":"JailbreakBench (source organisation)","released":"Dataset published on Hugging Face 2024-06","why_hard":"The benign twins are the point: a defence that simply refuses more often scores worse, so the benchmark measures discrimination rather than caution.","related":["truthfulqa","agentharm","harmbench"],"scores":[]}},"facts":null},{"benchmark":{"slug":"agentharm","formal_name":"AgentHarm","category":"Safety / Agents","effective_mode":"metadata_only","profile":{"resources":[{"kind":"dataset","url":"https://huggingface.co/datasets/ai-safety-institute/AgentHarm","label":"Source dataset"},{"kind":"issues","url":"https://huggingface.co/datasets/ai-safety-institute/AgentHarm/discussions","label":"Dataset discussions"},{"kind":"paper","url":"https://arxiv.org/abs/2410.09024","label":"arXiv:2410.09024"}],"task_format":"Multi-step agent tasks with a set of synthetic tools, each paired with a benign counterpart.","scoring":"Scored by a grading function per behaviour against the agent transcript; nothing is imported here.","metric":"harm score and refusal rate, with the benign pairs measuring over-refusal","size":"110 harmful and 110 benign behaviours in the source; none imported.","answer_access":"No problem text is imported, so there is no answer storage for this entry. The source distributes the answers under its own terms.","license":"other (declared on the Hugging Face dataset card)","history":"AgentHarm was introduced in 2024 to evaluate whether tool-using language model agents would carry out harmful requests over multiple steps. The AI Safety Institute distributes the dataset, whose Hugging Face publication is recorded as October 2024. Its design pairs harmful tasks with benign counterparts to distinguish resistance to harmful requests from indiscriminate refusal.","construction":"The source describes 110 harmful behaviours and 110 matched benign behaviours, presented as multi-step tasks using synthetic tools. Each behaviour has a grading function that checks the agent transcript. The pairing supports assessment of harmful completion alongside performance on related legitimate tasks.","limitations":"Synthetic tools provide a controlled setting, but successful completion does not establish that an agent could cause the corresponding harm in a live environment. Results depend on the tools, attack prompts and trajectory grading used in an evaluation. Refusal rates need to be read alongside benign-task performance, since refusing everything can suppress harmful completion without demonstrating useful discrimination.","saturation":"","citation":"@article{andriushchenko2024agentharm,\n  title={AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents},\n  author={Maksym Andriushchenko and Alexandra Souly and Mateusz Dziemian and Derek Duenas and Maxwell Lin and Justin Wang and Dan Hendrycks and Andy Zou and Zico Kolter and Matt Fredrikson and Eric Winsor and Jerome Wynne and Yarin Gal and Xander Davies},\n  journal={arXiv preprint arXiv:2410.09024},\n  year={2024}\n}","maintainer":"ai-safety-institute (source organisation)","released":"Dataset published on Hugging Face 2024-10","why_hard":"A single refusal is not enough: the agent has to keep refusing across a multi-step trajectory, and a jailbreak that survives one step often carries through the rest.","related":["jailbreakbench","truthfulqa","harmbench"],"scores":[]}},"facts":null},{"benchmark":{"slug":"critpt","formal_name":"CritPt","category":"Science","effective_mode":"full","profile":{"resources":[{"kind":"dataset","url":"https://huggingface.co/datasets/CritPt-Benchmark/CritPt","label":"Source dataset"},{"kind":"issues","url":"https://huggingface.co/datasets/CritPt-Benchmark/CritPt/discussions","label":"Dataset discussions"},{"kind":"site","url":"https://critpt.com/","label":"Official site"},{"kind":"paper","url":"https://arxiv.org/abs/2509.26574","label":"arXiv:2509.26574"}],"task_format":"Research-level physics problem descriptions with code templates. The import accepts scalar problem_description and code_template fields.","scoring":"For the 70 imported problems, the source ships an answer with every problem; this catalogue keeps it in restricted storage and never renders it. The authors grade with an automated pipeline built for physics-specific answer formats, and report average accuracy over five runs of the 70 challenges.","metric":"accuracy (README: average accuracy over 5 runs × 70 challenges)","size":"70 public challenges; catalog selection expects 70 task records from train, with a source-row limit of 200.","answer_access":"The import separates non-prompt fields into restricted answer storage when supplied; the converter alone does not establish whether usable answers are available.","license":"apache-2.0 (declared on the Hugging Face dataset card)","history":"The CritPt-Benchmark group published the CritPt dataset on Hugging Face in September 2025. It evaluates scientific understanding through research-level physics challenges that require multi-step reasoning and computation, rather than only recall of scientific facts.","construction":"","limitations":"CritPt covers research-level physics, so its results do not by themselves establish scientific competence across other disciplines or in experimental work. The public set contains 70 challenges, making coverage of the field dependent on a relatively small collection of problems. The authors report average accuracy over five runs using an automated pipeline for physics-specific answer formats; comparisons need to preserve that grading procedure and repetition policy.","saturation":"","citation":"@article{zhu2025probing,\n  title={Probing the Critical Point (CritPt) of AI Reasoning: a Frontier Physics Research Benchmark},\n  author={Zhu, Minhui and Tian, Minyang and Yang, Xiaocheng and Zhou, Tianci and Zhu, Penghao and Chertkov, Eli and Liu, Shengyan and Du, Yufeng and Yuan, Lifan and Ji, Ziming and others},\n  journal={arXiv preprint arXiv:2509.26574},\n  year={2025}\n}","maintainer":"CritPt-Benchmark (Hugging Face organisation)","released":"Dataset published on Hugging Face 2025-09","why_hard":"Requires scientific understanding, multi-step reasoning, and computation on research-level physics problems.","related":["frontierscience","scicode","olympiadbench"],"scores":[]}},"facts":{"slug":"critpt","task_count":70,"image_count":0,"formats":[{"count":70,"value":"text"}],"answers":[{"count":70,"value":"published"}],"configs":[{"count":70,"value":""}],"splits":[{"count":70,"value":"train"}],"waiting_count":70,"post_count":0,"discussed_count":0,"releases":[{"acquired":"2026-09-09","revision":"9b9fc8498596ec08ab5437a72f4aa18beef2b876","release_id":"9dfa7f3eeb6834c65f51084c27c40fbcbceeb26b745917b164eeb66685d68940"},{"acquired":"2026-09-09","revision":"9b9fc8498596ec08ab5437a72f4aa18beef2b876","release_id":"f82c4abb3348cce93c5af16f1d364aaa75ffa26812565b5fd254562cfe7ef090"},{"acquired":"2026-09-09","revision":"9b9fc8498596ec08ab5437a72f4aa18beef2b876","release_id":"b31f5bee2d6ed02213b1189d6bcfb041c908b9613927941b2ffc4ee78a268aee"},{"acquired":"2026-09-09","revision":"9b9fc8498596ec08ab5437a72f4aa18beef2b876","release_id":"ccb2286bf0ebfe2c3b849216af944f2f2478eba6e0cc96515a6d8d2ba8b7a18f"}],"terms":{"license":null,"checked_at":null,"distribution_terms":null,"publisher_requests":null}}},{"benchmark":{"slug":"frontierscience","formal_name":"FrontierScience","category":"Science","effective_mode":"full","profile":{"resources":[{"kind":"dataset","url":"https://huggingface.co/datasets/openai/frontierscience","label":"Source dataset"},{"kind":"issues","url":"https://huggingface.co/datasets/openai/frontierscience/discussions","label":"Dataset discussions"}],"task_format":"Scientific problems; the selected olympiad import accepts scalar problem and subject fields.","scoring":"For the 100 imported problems, the source ships an answer with every problem; this catalogue keeps it in restricted storage and never renders it.","metric":"","size":"Catalog selection: olympiad / test, expected 100 task records, source-row limit 200.","answer_access":"Fields outside the problem and subject prompt columns are kept in restricted storage when supplied; the converter alone does not establish whether usable answers are available.","license":"apache-2.0 (declared on the Hugging Face dataset card)","history":"OpenAI published the FrontierScience dataset on Hugging Face in December 2025 to evaluate expert-level scientific problem solving. Its public data separates olympiad problems from research problems, allowing performance on the two kinds of scientific work to be examined independently.","construction":"","limitations":"The catalogue selection covers 100 olympiad test records rather than the research portion of FrontierScience. Performance on that selection therefore does not establish performance on the research tasks. Solving written scientific problems also does not directly measure experimental execution or the validation of a new scientific finding.","saturation":"","citation":"","maintainer":"openai (Hugging Face organisation)","released":"Dataset published on Hugging Face 2025-12","why_hard":"Requires solving expert-level scientific tasks.","related":["critpt","scicode","olympiadbench"],"scores":[]}},"facts":{"slug":"frontierscience","task_count":100,"image_count":0,"formats":[{"count":100,"value":"text"}],"answers":[{"count":100,"value":"published"}],"configs":[{"count":100,"value":"olympiad"}],"splits":[{"count":100,"value":"test"}],"waiting_count":100,"post_count":0,"discussed_count":0,"releases":[{"acquired":"2026-09-09","revision":"25ed67db7da8f4591484e764008ff585544f5a30","release_id":"786d4ab34bb0e43d371ff0dcba018f2e67fa602bd48c987f3d64e758f9da44ca"},{"acquired":"2026-09-09","revision":"25ed67db7da8f4591484e764008ff585544f5a30","release_id":"04c1630d70990438eab9aa888c13d211f8d58245c8453a41cdc95b77acc63960"},{"acquired":"2026-09-09","revision":"25ed67db7da8f4591484e764008ff585544f5a30","release_id":"58981f75b05c7a1b68a7f504896cfba38e3bd0eea4b6598ce57de628df4683c9"},{"acquired":"2026-09-09","revision":"25ed67db7da8f4591484e764008ff585544f5a30","release_id":"b16673084a412dabaf5234bf99bfedc6924d0fbf9b1f3bad07a9032bc9d251cf"}],"terms":{"license":null,"checked_at":null,"distribution_terms":null,"publisher_requests":null}}},{"benchmark":{"slug":"gpqa","formal_name":"GPQA","category":"Science","effective_mode":"metadata_only","profile":{"resources":[{"kind":"dataset","url":"https://huggingface.co/datasets/Idavidrein/gpqa","label":"Source dataset"},{"kind":"issues","url":"https://huggingface.co/datasets/Idavidrein/gpqa/discussions","label":"Dataset discussions"},{"kind":"paper","url":"https://arxiv.org/abs/2311.12022","label":"arXiv:2311.12022"}],"task_format":"Graduate-level multiple-choice questions in biology, physics and chemistry.","scoring":"Access is gated and the authors ask that the questions not be propagated in plain text, to keep them out of training data. This catalogue links to the official distribution instead.","metric":"","size":"Not imported here; see the official distribution for the current size.","answer_access":"Access is gated and the authors ask that the questions not be propagated in plain text, to keep them out of training data. This catalogue links to the official distribution instead.","license":"cc-by-4.0 (declared on the source)","history":"David Rein and collaborators introduced GPQA in 2023 to test expert scientific knowledge that could not readily be substituted by web search. The benchmark focuses on graduate-level biology, physics and chemistry. Its motivation was to construct questions that domain experts could answer but skilled non-experts would struggle with even when allowed to search online.","construction":"Domain experts wrote the multiple-choice questions, and the collection process included expert validation and attempts by non-experts with web access. This design checks both specialist answerability and resistance to straightforward search. The Diamond subset selects questions using the validation results and is a distinct evaluation slice rather than another name for the whole dataset.","limitations":"GPQA measures selection among supplied answers in three scientific fields, rather than experimental ability or the production of independently verified research. Results on Diamond should not be conflated with results on other subsets. The authors restrict access and ask that questions not be propagated as plain text to reduce training contamination; those precautions do not establish that any particular model has never encountered the material.","saturation":"","citation":"","maintainer":"Idavidrein (source organisation)","released":"Dataset published on Hugging Face 2023-11","why_hard":"Written and validated so that skilled non-experts with unrestricted web access still score near chance.","related":["critpt","frontierscience","olympiadbench"],"scores":[]}},"facts":null},{"benchmark":{"slug":"scicode","formal_name":"SciCode","category":"Science / Code","effective_mode":"full","profile":{"resources":[{"kind":"dataset","url":"https://huggingface.co/datasets/SciCode1/SciCode","label":"Source dataset"},{"kind":"issues","url":"https://huggingface.co/datasets/SciCode1/SciCode/discussions","label":"Dataset discussions"},{"kind":"paper","url":"https://arxiv.org/abs/2407.13168","label":"arXiv:2407.13168"}],"task_format":"Parent scientific problems contain descriptions, background, input/output information, and dependencies. sub_steps become linked child tasks with step descriptions, background, input/output information, dependencies, and code templates when supplied.","scoring":"For the 65 imported problems, the source ships an answer with every problem; this catalogue keeps it in restricted storage and never renders it.","metric":"","size":"Catalog dev selection: 15 parent problems and 50 subproblems, totaling 65 task records.","answer_access":"Parent general_solution and child step_solution, ground_truth_code, and test_cases fields, when supplied, are kept outside the prompt in restricted records.","license":"apache-2.0 (declared on the Hugging Face dataset card)","history":"","construction":"SciCode structures scientific coding problems as parent tasks with linked subproblems. Records provide descriptions, scientific background, input and output information, dependencies and, when supplied, code templates. The catalogue development selection contains 15 parent problems and 50 subproblems, giving 65 task records rather than 65 independent research problems. Reference solutions and test fields are kept outside the displayed prompts.","limitations":"Solving a supplied scientific coding task does not establish the ability to choose a research question or validate a scientific model against experimental evidence. Performance on an individual subproblem also does not show that a system can combine all the steps into a working parent solution. Results on this development selection should be distinguished from held-out evaluation, and parent and child outcomes should not be treated as independent observations.","saturation":"","citation":"","maintainer":"SciCode1 (Hugging Face organisation)","released":"Dataset published on Hugging Face 2025-02","why_hard":"Requires solving scientific research problems through code, with parent problems decomposed into subproblems.","related":["critpt","frontierscience","hlce"],"scores":[]}},"facts":{"slug":"scicode","task_count":65,"image_count":0,"formats":[{"count":15,"value":"scicode"},{"count":50,"value":"scicode-step"}],"answers":[{"count":65,"value":"published"}],"configs":[{"count":65,"value":""}],"splits":[{"count":65,"value":"dev"}],"waiting_count":65,"post_count":0,"discussed_count":0,"releases":[{"acquired":"2026-09-09","revision":"4510f6a6aa27c43fad7b43da2c59602a86e88480","release_id":"1993c24866359ce785e35edf5addacd53d15cee7fc3f76ccb7fe604139a52502"},{"acquired":"2026-09-09","revision":"4510f6a6aa27c43fad7b43da2c59602a86e88480","release_id":"25290d9a506ea1a294bf866fc8affad23cc2254ef575b67948da79a594942709"},{"acquired":"2026-09-09","revision":"4510f6a6aa27c43fad7b43da2c59602a86e88480","release_id":"502aa46ac3e3342f5b36834a7b61a9a9c9b64bb6d2d9e687cd15e9b385ec6a78"},{"acquired":"2026-09-09","revision":"4510f6a6aa27c43fad7b43da2c59602a86e88480","release_id":"8507fdfa666112f80dc8e95b2c58834dec2d52942faf811634f3d242c293b997"},{"acquired":"2026-09-08","revision":"4510f6a6aa27c43fad7b43da2c59602a86e88480","release_id":"fb2b7319752658f233e8801750d5cbae802d6b5c1833db836fe56f721eb79596"}],"terms":{"license":null,"checked_at":null,"distribution_terms":null,"publisher_requests":null}}}]}