BigCodeBench poses 1,140 function-level tasks drawn across 139 libraries. It tests whether a model can compose several real APIs correctly rather than write one self-contained function.
Full text1,139 tasks
Metric
Pass@1 (the official runner also reports Pass@5 and Pass@10)
Size
1,139 unique problems imported from the default / v0.1.0_hf selection of 1,140 rows; 1 row repeats an earlier prompt and is recorded as a variant. Later version splits repeat the same task IDs and are not imported separately.
License
apache-2.0 (declared on the Hugging Face dataset card)
CodeContests is the competitive-programming dataset built for AlphaCode, drawing problems from Codeforces and similar sites. Each problem ships with both correct and incorrect submissions and a large generated test suite…
Size
Roughly 13,500 problems in the source; none imported.
License
cc-by-4.0 (declared on the Hugging Face dataset card)
CRUXEval takes 800 short Python functions and asks either for the output given an input (CRUXEval-O) or for an input that produces a given output (CRUXEval-I). It measures the ability to execute code mentally rather than…
Full text800 tasks
Metric
pass@1 on output prediction and on input prediction, reported separately
Size
800 problems imported from the default / test selection (the full split).
DS-1000 builds 1,000 data-science problems from real StackOverflow questions across seven libraries including NumPy, Pandas and Matplotlib. The problems are perturbed so that recalling the original answer does not solve …
Full text1,000 tasks
Metric
accuracy (the source paper reports the best public system at 43.3%)
Size
1,000 problems imported from the default / test selection (the full split).
License
cc-by-sa-4.0 (declared on the Hugging Face dataset card)
Humanity's Last Code Exam evaluates problem solving and code generation using ICPC World Finals and IOI problems. The project describes 235 problems; this import selects its public ICPC dataset of 146 problems.
Full text146 tasks
Size
235 problems described in the catalog introduction; this selection expects 146 ICPC problem records from ICPC-World-Finals / examples, with a source-row limit of 200.
HumanEval asks for a Python function body from a docstring and signature, checked by hidden unit tests. It is the original functional-correctness benchmark for code generation.
Full text164 tasks
Metric
pass@1 (unbiased estimator over sampled completions)
Size
164 problems imported from the openai_humaneval / test selection, with a source-row limit of 200.
HumanEval+ keeps the 164 original HumanEval problems and multiplies their tests by roughly 80×. It exists because the original suite was loose enough to pass implementations that were actually wrong.
Full text164 tasks
Metric
pass@1 on the expanded tests
Size
All 164 problems imported from the default / test selection, with a source-row limit of 200.
License
apache-2.0 (declared on the Hugging Face dataset card)
LiveCodeBench continuously collects new competitive-programming problems to evaluate coding capabilities. Its initial release_v1 contains 400 problems, with source platform, contest date, and dataset version tracked expl…
Full text100 tasks
Size
400 problems in initial release_v1 described in the catalog introduction; this release_v1 / test selection expects 100 task records, limited to the first 100 source rows.
Mostly Basic Python Problems asks for a short Python function from a one-sentence description, checked by assertions. The sanitized split is the hand-verified subset.
Full text1,401 tasks
Metric
pass@1 on the assertion tests
Size
1,401 problems imported: the full config across its four splits (974 records) and the hand-verified sanitized config across its four splits (427 records). The two configs overlap by upstream task_id and are kept as separate variant pages.
MBPP+ takes the 378 problems of the sanitized MBPP set and multiplies their tests by roughly 35 times. It removes the same test-weakness inflation that HumanEval+ addresses.
Full text378 tasks
Metric
pass@1 on the expanded tests
Size
378 problems imported from the default / test selection (the full split).
License
apache-2.0 (declared on the Hugging Face dataset card)