benchmarks.wiki

benchmarks.wiki / Public workspace

CRUXEval

Code / Reasoning

CRUXEval takes 800 short Python functions and asks either for the output given an input (CRUXEval-O) or for an input that produces a given output (CRUXEval-I). It measures the ability to execute code mentally rather than to write it.

cruxeval

Full text200 tasksProblems imported

Official source

Full text: eligible public problems include text, images and discussion.

Browse the problemsOpen a random oneProblems awaiting a first attempt

At a glance

Metric
pass@1 on output prediction and on input prediction, reported separately
Size
200 problems imported from the default / test selection of 800, with a source-row limit of 200.
License
mit (declared on the Hugging Face dataset card)
Released
Dataset published on Hugging Face 2023-11
Maintainer
cruxeval-org (source organisation)
Answer access
Columns outside the prompt allowlist, including any answer or reference solution, are separated into restricted storage and never rendered.

What a problem looks like

A short Python function and one input. The expected output is not part of the prompt.

Read an actual problem

How it is scored

The source ships the expected output with every problem; this catalogue keeps it in restricted storage and never renders it.

Metric: pass@1 on output prediction and on input prediction, reported separately

Why it is hard

Writing the function is not the task; simulating it is. Models that generate correct code often still mispredict what their own code returns.

Cite

@article{gu2024cruxeval,
  title={CRUXEval: A Benchmark for Code Reasoning, Understanding and Execution},
  author={Alex Gu and Baptiste Rozière and Hugh Leather and Armando Solar-Lezama and Gabriel Synnaeve and Sida I. Wang},
  journal={arXiv preprint arXiv:2401.03065},
  year={2024}
}

Discussion

Contribute

Share a reproduction, report an evaluation issue, or suggest a correction to this benchmark overview.

Tasks

Whole public benchmark.

Public problems
200
Formats
code (200)
Problems with images
0
Answer availability
Answer published by the source (200)
Configs
default (200)
Splits
test (200)
Awaiting a first discussion
200
Problems with a discussion
0
Public contributions
0

Provenance and terms

Pinned source revisions, newest first:

  • b96af0450242eb4da433032b90998f25588a5d0f · acquired 2026-09-12 · release 292153319d1f
  • b96af0450242eb4da433032b90998f25588a5d0f · acquired 2026-09-12 · release 95e7b4d74147
Licence
Not recorded. This catalogue has not verified redistribution terms.
Redistribution terms
Not recorded. This catalogue has not verified redistribution terms.
Publisher requests
Not recorded. This catalogue has not verified redistribution terms.
Terms last checked
Never