benchmarks.wiki

benchmarks.wiki / Public workspace

BigCodeBench

Code

BigCodeBench poses 1,140 function-level tasks drawn across 139 libraries. It tests whether a model can compose several real APIs correctly rather than write one self-contained function.

bigcodebench

Full text200 tasksProblems imported

Official source

Full text: eligible public problems include text, images and discussion.

Browse the problemsOpen a random oneProblems awaiting a first attempt

At a glance

Metric
Pass@1 (the official runner also reports Pass@5 and Pass@10)
Size
200 problems imported from the default / v0.1.0_hf selection of 1,140, with a source-row limit of 200.
License
apache-2.0 (declared on the Hugging Face dataset card)
Released
Dataset published on Hugging Face 2024-06
Maintainer
bigcode (source organisation)
Answer access
Columns outside the prompt allowlist, including any answer or reference solution, are separated into restricted storage and never rendered.

What a problem looks like

Two forms of the same task — a complete docstring prompt and an instruction-style prompt — plus the code skeleton, the entry point, the libraries involved and the structured docstring.

Read an actual problem

How it is scored

Correctness is decided by running a unittest suite against the candidate, so the reference solution the source ships is not a graded answer; it stays in restricted storage.

Metric: Pass@1 (the official runner also reports Pass@5 and Pass@10)

Why it is hard

The libraries have to be used as they actually behave, not as they are usually summarised. Most failures are correct-looking calls with the wrong arguments or the wrong return handling.

Cite

@article{zhuo2024bigcodebench,
  title={BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions},
  author={Terry Yue Zhuo and Minh Chien Vu and Jenny Chim and Han Hu and Wenhao Yu and Ratnadira Widyasari and Imam Nur Bani Yusuf and Haolan Zhan and Junda He and Indraneil Paul and Simon Brunner and Chen Gong and Thong Hoang and Armel Randy Zebaze and Xiaoheng Hong and Wen-Ding Li and Jean Kaddour and Ming Xu and Zhihan Zhang and Prateek Yadav and Naman Jain and Alex Gu and Zhoujun Cheng and Jiawei Liu and Qian Liu and Zijian Wang and Binyuan Hui and Niklas Muennighoff and David Lo and Daniel Fried and Xiaoning Du and Harm de Vries and Leandro Von Werra},
  journal={arXiv preprint arXiv:2406.15877},
  year={2024}
}

Discussion

Contribute

Share a reproduction, report an evaluation issue, or suggest a correction to this benchmark overview.

Tasks

Whole public benchmark.

Public problems
200
Formats
code (200)
Problems with images
0
Answer availability
Answer published by the source (200)
Configs
default (200)
Splits
v0.1.0_hf (200)
Awaiting a first discussion
200
Problems with a discussion
0
Public contributions
0

Provenance and terms

Pinned source revisions, newest first:

  • b74c0d0bf70d2c0bc459be537895cca163007f1a · acquired 2026-09-12 · release 7fd659a6a430
  • b74c0d0bf70d2c0bc459be537895cca163007f1a · acquired 2026-09-12 · release 1b4e12bf295f
Licence
Not recorded. This catalogue has not verified redistribution terms.
Redistribution terms
Not recorded. This catalogue has not verified redistribution terms.
Publisher requests
Not recorded. This catalogue has not verified redistribution terms.
Terms last checked
Never