Discussion
0 posts
No posts in this thread yet.
benchmarks.wiki / Public workspace
HumanEval
Code
HumanEval asks for a Python function body from a docstring and signature, checked by hidden unit tests. It is the original functional-correctness benchmark for code generation.
humaneval
Full text: eligible public problems include text, images and discussion.
History
OpenAI introduced HumanEval in 2021 as part of its evaluation of Codex code-generation models. It made executable functional correctness a central measure of whether a model could complete a Python function from its specification. The March 2022 Hugging Face date recorded here refers to that distribution, not the original release.
How it was built
HumanEval contains 164 manually written Python programming problems, each with a function signature, a docstring, a reference implementation and unit tests. A model supplies the function completion, and the evaluator runs it against the tests. Pass rates measure successful execution rather than textual similarity to the reference implementation.
At a glance
What a problem looks like
A Python function signature and docstring in prompt, with the entry point named. The reference solution and the tests are kept out of the prompt.
How it is scored
For the 164 imported problems the source ships a reference implementation and a unit-test suite; this catalogue keeps both in restricted storage and never renders them. Correctness is decided by executing a candidate against the tests, not by comparing it with the reference.
Metric: pass@1 (unbiased estimator over sampled completions)
Reported scores
Evaluation conditions differ between rows, so this is not a ranking. Rows marked vendor are reports by the model’s own organization.
| Model | Score | Metric | Setup | Reported by | Source |
|---|---|---|---|---|---|
| DeepSeek-V3DeepSeek | 65.2 | Pass@1 | Base model; 0-shot. | vendor (self-reported) | Base Model, Standard Benchmarks, HumanEval (Pass@1), final column |
Why it is hard
Correctness is decided by execution, not by resemblance to a reference: a plausible-looking function that fails one edge case scores zero.
Limitations and criticism
The small set of standalone Python functions does not measure repository navigation, dependency management or extended maintenance work. The original test suites can miss incorrect behaviour on untested inputs, so passing them is weaker than demonstrating general correctness. Public availability creates a risk of training exposure, while sampling settings and the number of allowed completions affect reported pass rates.
How much room is left
The recorded vendor result for DeepSeek-V3 base is 65.2 percent pass@1 with no examples in the prompt, leaving headroom for that model and protocol. This single base-model result cannot establish overall saturation among specialised or instruction-tuned coding systems. With only 164 tasks, a small number of changed outcomes can noticeably move the aggregate score.
Reference links
Cite
@misc{chen2021evaluating,
title={Evaluating Large Language Models Trained on Code},
author={Mark Chen and Jerry Tworek and Heewoo Jun and Qiming Yuan and Henrique Ponde de Oliveira Pinto and Jared Kaplan and Harri Edwards and Yuri Burda and Nicholas Joseph and Greg Brockman and Alex Ray and Raul Puri and Gretchen Krueger and Michael Petrov and Heidy Khlaaf and Girish Sastry and Pamela Mishkin and Brooke Chan and Scott Gray and Nick Ryder and Mikhail Pavlov and Alethea Power and Lukasz Kaiser and Mohammad Bavarian and Clemens Winter and Philippe Tillet and Felipe Petroski Such and Dave Cummings and Matthias Plappert and Fotios Chantzis and Elizabeth Barnes and Ariel Herbert-Voss and William Hebgen Guss and Alex Nichol and Alex Paino and Nikolas Tezak and Jie Tang and Igor Babuschkin and Suchir Balaji and Shantanu Jain and William Saunders and Christopher Hesse and Andrew N. Carr and Jan Leike and Josh Achiam and Vedant Misra and Evan Morikawa and Alec Radford and Matthew Knight and Miles Brundage and Mira Murati and Katie Mayer and Peter Welinder and Bob McGrew and Dario Amodei and Sam McCandlish and Ilya Sutskever and Wojciech Zaremba},
year={2021},
eprint={2107.03374},
archivePrefix={arXiv},
primaryClass={cs.LG}
}Discussion
1 thread and 0 posts on this page. Anyone may reply; no account is required.
Contribute
Share a reproduction, report an evaluation issue, or suggest a correction to this benchmark overview.
Tasks
Whole public benchmark.
- Public problems
- 164
- Formats
- text (164)
- Problems with images
- 0
- Answer availability
- Answer published by the source (164)
- Configs
- openai_humaneval (164)
- Splits
- test (164)
- Awaiting a first discussion
- 164
- Problems with a discussion
- 0
- Public contributions
- 0
Provenance and terms
Pinned source revisions, newest first:
7dce6050a7d6d172f3cc5c32aa97f52fa1a2e544· acquired 2026-09-11 · release29b40586e9ce7dce6050a7d6d172f3cc5c32aa97f52fa1a2e544· acquired 2026-09-11 · release09c39d799d2c7dce6050a7d6d172f3cc5c32aa97f52fa1a2e544· acquired 2026-09-11 · release143d6b426d5d7dce6050a7d6d172f3cc5c32aa97f52fa1a2e544· acquired 2026-09-11 · release2bbe60b1260b
- FIX = """
Answer published by the source
- def count_nums(arr):
Answer published by the source
- def correct_bracketing(brackets: str):
Answer published by the source
- def sort_third(l: list):
Answer published by the source
- from typing import List
Answer published by the source
- def solution(lst):
Answer published by the source
- def below_threshold(l: list, t: int):
Answer published by the source
- def fix_spaces(text):
Answer published by the source
- def largest_smallest_integers(lst):
Answer published by the source
- def sort_array(arr):
Answer published by the source
- def flip_case(string: str) -> str:
Answer published by the source
- import math
Answer published by the source
- def tri(n):
Answer published by the source
- def solve(s):
Answer published by the source
- def triangle_area(a, b, c):
Answer published by the source
- def words_in_sentence(sentence):
Answer published by the source
- from typing import List
Answer published by the source
- from typing import List
Answer published by the source
- def fruit_distribution(s,n):
Answer published by the source
- def sort_even(l: list):
Answer published by the source
- def get_closest_vowel(word):
Answer published by the source
- from typing import List
Answer published by the source
- def pairs_sum_to_zero(l):
Answer published by the source
- def derivative(xs: list):
Answer published by the source
- def median(l: list):
Answer published by the source
- from typing import List
Answer published by the source
- def check_dict_case(dict):
Answer published by the source
- from typing import List
Answer published by the source
- def add_elements(arr, k):
Answer published by the source
- def hex_key(num):
Answer published by the source
- def iscube(a):
Answer published by the source
- def numerical_letter_grade(grades):
Answer published by the source
- def rounded_avg(n, m):
Answer published by the source
- def is_bored(S):
Answer published by the source
- def prime_fib(n: int):
Answer published by the source
- def how_many_times(string: str, substring: str) -> int:
Answer published by the source
- def right_angle_triangle(a, b, c):
Answer published by the source
- def is_nested(string):
Answer published by the source
- def will_it_fly(q,w):
Answer published by the source
- from typing import List
Answer published by the source
- def prime_length(string):
Answer published by the source
- from typing import List
Answer published by the source
- def generate_integers(a, b):
Answer published by the source
- def string_to_md5(text):
Answer published by the source
- def max_element(l: list):
Answer published by the source
- def is_happy(s):
Answer published by the source
- def encode_shift(s: str):
Answer published by the source
- def unique_digits(x):
Answer published by the source
- def get_positive(l: list):
Answer published by the source
- def strlen(string: str) -> int:
Answer published by the source