benchmarks.wiki

benchmarks.wiki / Public workspace

CritPt

Science

CritPt evaluates scientific understanding and multi-step reasoning and computation on research-level physics problems. Its public dataset contains 70 challenges with problem descriptions and code templates.

critpt

Full text70 tasks

Official source

Full text: eligible public problems include text, images and discussion.

Download answersBrowse the problemsAnswersOpen a random oneProblems awaiting a first attempt

History

The CritPt-Benchmark group published the CritPt dataset on Hugging Face in September 2025. It evaluates scientific understanding through research-level physics challenges that require multi-step reasoning and computation, rather than only recall of scientific facts.

At a glance

Metric
accuracy (README: average accuracy over 5 runs × 70 challenges)
Size
70 public challenges; catalog selection expects 70 task records from train, with a source-row limit of 200.
License
apache-2.0 (declared on the Hugging Face dataset card)
Released
Dataset published on Hugging Face 2025-09
Maintainer
CritPt-Benchmark (Hugging Face organisation)
Answer access
The import separates non-prompt fields into restricted answer storage when supplied; the converter alone does not establish whether usable answers are available.

What a problem looks like

Research-level physics problem descriptions with code templates. The import accepts scalar problem_description and code_template fields.

Read an actual problem

How it is scored

For the 70 imported problems, the source ships an answer with every problem; this catalogue keeps it in restricted storage and never renders it. The authors grade with an automated pipeline built for physics-specific answer formats, and report average accuracy over five runs of the 70 challenges.

Metric: accuracy (README: average accuracy over 5 runs × 70 challenges)

Why it is hard

Requires scientific understanding, multi-step reasoning, and computation on research-level physics problems.

Limitations and criticism

CritPt covers research-level physics, so its results do not by themselves establish scientific competence across other disciplines or in experimental work. The public set contains 70 challenges, making coverage of the field dependent on a relatively small collection of problems. The authors report average accuracy over five runs using an automated pipeline for physics-specific answer formats; comparisons need to preserve that grading procedure and repetition policy.

Cite

@article{zhu2025probing,
  title={Probing the Critical Point (CritPt) of AI Reasoning: a Frontier Physics Research Benchmark},
  author={Zhu, Minhui and Tian, Minyang and Yang, Xiaocheng and Zhou, Tianci and Zhu, Penghao and Chertkov, Eli and Liu, Shengyan and Du, Yufeng and Yuan, Lifan and Ji, Ziming and others},
  journal={arXiv preprint arXiv:2509.26574},
  year={2025}
}

Discussion

1 thread and 0 posts on this page. Anyone may reply; no account is required.

Contribute

Share a reproduction, report an evaluation issue, or suggest a correction to this benchmark overview.

Tasks

Whole public benchmark.

Public problems
70
Formats
text (70)
Problems with images
0
Answer availability
Answer published by the source (70)
Configs
Default (70)
Splits
train (70)
Awaiting a first discussion
70
Problems with a discussion
0
Public contributions
0

Provenance and terms

Pinned source revisions, newest first:

  • 9b9fc8498596ec08ab5437a72f4aa18beef2b876 · acquired 2026-09-09 · release 9dfa7f3eeb68
  • 9b9fc8498596ec08ab5437a72f4aa18beef2b876 · acquired 2026-09-09 · release f82c4abb3348
  • 9b9fc8498596ec08ab5437a72f4aa18beef2b876 · acquired 2026-09-09 · release b31f5bee2d6e
  • 9b9fc8498596ec08ab5437a72f4aa18beef2b876 · acquired 2026-09-09 · release ccb2286bf0eb
Licence
Not recorded. This catalogue has not verified redistribution terms.
Redistribution terms
Not recorded. This catalogue has not verified redistribution terms.
Publisher requests
Not recorded. This catalogue has not verified redistribution terms.
Terms last checked
Never