benchmarks.wiki

benchmarks.wiki / Public workspace

BigCodeBench

Code

BigCodeBench poses 1,140 function-level tasks drawn across 139 libraries. It tests whether a model can compose several real APIs correctly rather than write one self-contained function.

bigcodebench

Full text1,139 tasks

Official source

Full text: eligible public problems include text, images and discussion.

Download answersBrowse the problemsAnswersOpen a random oneProblems awaiting a first attempt

History

The BigCode project introduced BigCodeBench in 2024 to evaluate code generation that combines practical library functionality. It addressed a gap left by benchmarks dominated by short, self-contained functions. The dataset distribution recorded in this catalogue was published in June 2024.

How it was built

BigCodeBench was developed through model-assisted task creation and human review of the resulting programming tasks. It contains 1,140 function-level tasks covering 139 libraries, with reference implementations and executable unit tests. The same tasks have a detailed docstring form and an instruction-style form, and correctness is evaluated by running candidate code against the tests.

At a glance

Metric
Pass@1 (the official runner also reports Pass@5 and Pass@10)
Size
1,139 unique problems imported from the default / v0.1.0_hf selection of 1,140 rows; 1 row repeats an earlier prompt and is recorded as a variant. Later version splits repeat the same task IDs and are not imported separately.
License
apache-2.0 (declared on the Hugging Face dataset card)
Released
Dataset published on Hugging Face 2024-06
Maintainer
bigcode (source organisation)
Answer access
Columns outside the prompt allowlist, including any answer or reference solution, are separated into restricted storage and never rendered.

What a problem looks like

Two forms of the same task — a complete docstring prompt and an instruction-style prompt — plus the code skeleton, the entry point, the libraries involved and the structured docstring.

Read an actual problem

How it is scored

Correctness is decided by running a unittest suite against the candidate, so the reference solution the source ships is not a graded answer; it stays in restricted storage.

Metric: Pass@1 (the official runner also reports Pass@5 and Pass@10)

Why it is hard

The libraries have to be used as they actually behave, not as they are usually summarised. Most failures are correct-looking calls with the wrong arguments or the wrong return handling.

Limitations and criticism

Passing the supplied tests establishes agreement with their checks rather than correctness for every possible input. Library versions and the execution environment matter because tasks depend on external APIs. Function-level generation does not directly assess maintaining a large repository or negotiating ambiguous requirements. Results from the docstring and instruction variants should be distinguished, as should single-attempt and multiple-attempt pass rates.

Cite

@article{zhuo2024bigcodebench,
  title={BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions},
  author={Terry Yue Zhuo and Minh Chien Vu and Jenny Chim and Han Hu and Wenhao Yu and Ratnadira Widyasari and Imam Nur Bani Yusuf and Haolan Zhan and Junda He and Indraneil Paul and Simon Brunner and Chen Gong and Thong Hoang and Armel Randy Zebaze and Xiaoheng Hong and Wen-Ding Li and Jean Kaddour and Ming Xu and Zhihan Zhang and Prateek Yadav and Naman Jain and Alex Gu and Zhoujun Cheng and Jiawei Liu and Qian Liu and Zijian Wang and Binyuan Hui and Niklas Muennighoff and David Lo and Daniel Fried and Xiaoning Du and Harm de Vries and Leandro Von Werra},
  journal={arXiv preprint arXiv:2406.15877},
  year={2024}
}

Discussion

1 thread and 0 posts on this page. Anyone may reply; no account is required.

Contribute

Share a reproduction, report an evaluation issue, or suggest a correction to this benchmark overview.

Tasks

Whole public benchmark.

Public problems
1,139
Formats
code (1,139)
Problems with images
0
Answer availability
Answer published by the source (1,139)
Configs
default (1,139)
Splits
v0.1.0_hf (1,139)
Awaiting a first discussion
1,139
Problems with a discussion
0
Public contributions
0

Provenance and terms

Pinned source revisions, newest first:

  • b74c0d0bf70d2c0bc459be537895cca163007f1a · acquired 2026-09-12 · release 7fd659a6a430
  • b74c0d0bf70d2c0bc459be537895cca163007f1a · acquired 2026-09-12 · release 4ef02375478c
  • b74c0d0bf70d2c0bc459be537895cca163007f1a · acquired 2026-09-12 · release 1b4e12bf295f
Licence
Not recorded. This catalogue has not verified redistribution terms.
Redistribution terms
Not recorded. This catalogue has not verified redistribution terms.
Publisher requests
Not recorded. This catalogue has not verified redistribution terms.
Terms last checked
Never