benchmarks.wiki

benchmarks.wiki / Public workspace

CRUXEval

Code / Reasoning

CRUXEval takes 800 short Python functions and asks either for the output given an input (CRUXEval-O) or for an input that produces a given output (CRUXEval-I). It measures the ability to execute code mentally rather than to write it.

cruxeval

Full text800 tasks

Official source

Full text: eligible public problems include text, images and discussion.

Download answersBrowse the problemsAnswersOpen a random oneProblems awaiting a first attempt

History

The CRUXEval research group introduced the benchmark in a 2024 paper to separate code reasoning from code generation. Its two tasks ask a model either to predict a function's output or to find an input that yields a specified output. This catalogue records a November 2023 Hugging Face publication, preceding the paper.

How it was built

CRUXEval consists of 800 short Python functions with associated inputs and outputs. The functions were generated with a code model and filtered, with execution used to establish input-output consistency. Each function supports output prediction and input prediction, which are evaluated separately.

At a glance

Metric
pass@1 on output prediction and on input prediction, reported separately
Size
800 problems imported from the default / test selection (the full split).
License
mit (declared on the Hugging Face dataset card)
Released
Dataset published on Hugging Face 2023-11
Maintainer
cruxeval-org (source organisation)
Answer access
Columns outside the prompt allowlist, including any answer or reference solution, are separated into restricted storage and never rendered.
Highest recorded
69.8 Acc. — DeepSeek-V3 (vendor, self-reported)

What a problem looks like

A short Python function and one input. The expected output is not part of the prompt.

Read an actual problem

How it is scored

The source ships the expected output with every problem; this catalogue keeps it in restricted storage and never renders it.

Metric: pass@1 on output prediction and on input prediction, reported separately

Reported scores

Evaluation conditions differ between rows, so this is not a ranking. Rows marked vendor are reports by the model’s own organization.

ModelScoreMetricSetupReported bySource
DeepSeek-V3DeepSeek69.8Acc.Base model; 2-shot; output prediction.vendor (self-reported)Base Model, Standard Benchmarks, CRUXEval-O (Acc.), final column
DeepSeek-V3DeepSeek67.3Acc.Base model; 2-shot; input prediction.vendor (self-reported)Base Model, Standard Benchmarks, CRUXEval-I (Acc.), final column

Why it is hard

Writing the function is not the task; simulating it is. Models that generate correct code often still mispredict what their own code returns.

Limitations and criticism

Short, synthetic Python functions provide a narrow test of program behaviour rather than a measure of repository-level software engineering. Input prediction may admit multiple correct inputs, so correctness concerns the output produced rather than matching one reference input. Access to a Python interpreter changes the task substantially and needs to be distinguished from reasoning without execution.

How much room is left

The recorded vendor results for DeepSeek-V3 base are 69.8 percent on output prediction and 67.3 percent on input prediction, both with two examples in the prompt. These reports leave substantial headroom in each direction under that setup. They do not establish the present best result across all systems or tool-use protocols.

Cite

@article{gu2024cruxeval,
  title={CRUXEval: A Benchmark for Code Reasoning, Understanding and Execution},
  author={Alex Gu and Baptiste Rozière and Hugh Leather and Armando Solar-Lezama and Gabriel Synnaeve and Sida I. Wang},
  journal={arXiv preprint arXiv:2401.03065},
  year={2024}
}

Discussion

1 thread and 0 posts on this page. Anyone may reply; no account is required.

Contribute

Share a reproduction, report an evaluation issue, or suggest a correction to this benchmark overview.

Tasks

Whole public benchmark.

Public problems
800
Formats
code (800)
Problems with images
0
Answer availability
Answer published by the source (800)
Configs
default (800)
Splits
test (800)
Awaiting a first discussion
800
Problems with a discussion
0
Public contributions
0

Provenance and terms

Pinned source revisions, newest first:

  • b96af0450242eb4da433032b90998f25588a5d0f · acquired 2026-09-12 · release 95e7b4d74147
  • b96af0450242eb4da433032b90998f25588a5d0f · acquired 2026-09-12 · release 292153319d1f
  • b96af0450242eb4da433032b90998f25588a5d0f · acquired 2026-09-12 · release a5abcc0b205a
Licence
Not recorded. This catalogue has not verified redistribution terms.
Redistribution terms
Not recorded. This catalogue has not verified redistribution terms.
Publisher requests
Not recorded. This catalogue has not verified redistribution terms.
Terms last checked
Never