Discussion
0 posts
No posts in this thread yet.
benchmarks.wiki / Public workspace
HumanEval+
Code
HumanEval+ keeps the 164 original HumanEval problems and multiplies their tests by roughly 80×. It exists because the original suite was loose enough to pass implementations that were actually wrong.
humaneval-plus
Full text: eligible public problems include text, images and discussion.
History
The EvalPlus research group introduced HumanEval+ in 2023 to investigate whether code-generation benchmarks were accepting incorrect programs because their tests were too weak. It retained the original HumanEval tasks while expanding the checks applied to candidate implementations. The January 2024 publication date recorded here refers to the evalplus Hugging Face distribution.
How it was built
HumanEval+ retains all 164 Python function-completion problems from HumanEval and expands their tests by roughly a factor of 80. The EvalPlus approach combines model-generated test inputs with mutation-based generation to explore additional cases, using reference implementations to obtain expected outputs for valid inputs. A candidate is evaluated by executing its function against the expanded test suite, rather than by comparing its source text with a reference solution.
At a glance
What a problem looks like
A Python function signature with its docstring, plus the entry-point name. The reference implementation and the test suite are not part of the prompt.
How it is scored
Correctness is decided by executing the candidate against the expanded test suite, so the reference implementation the source ships is not a graded answer; it stays in restricted storage.
Metric: pass@1 on the expanded tests
Why it is hard
The problems are short, but the added tests probe the edge cases the original suite missed, so a solution that only handles the obvious path now fails.
Limitations and criticism
Expanded tests improve detection of incorrect implementations but do not prove correctness for every possible input. Evaluation still depends on the adequacy of task specifications, reference implementations and the particular test-suite version. The tasks are short, standalone Python functions and do not measure repository navigation, maintenance or integration with a larger software system. Reusing public HumanEval prompts also leaves potential prior exposure unresolved.
Reference links
Cite
@article{liu2023humanevalplus,
title={Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation},
author={Jiawei Liu and Chunqiu Steven Xia and Yuyao Wang and Lingming Zhang},
journal={arXiv preprint arXiv:2305.01210},
year={2023}
}Discussion
1 thread and 0 posts on this page. Anyone may reply; no account is required.
Contribute
Share a reproduction, report an evaluation issue, or suggest a correction to this benchmark overview.
Tasks
Whole public benchmark.
- Public problems
- 164
- Formats
- code (164)
- Problems with images
- 0
- Answer availability
- Answer published by the source (164)
- Configs
- default (164)
- Splits
- test (164)
- Awaiting a first discussion
- 164
- Problems with a discussion
- 0
- Public contributions
- 0
Provenance and terms
Pinned source revisions, newest first:
d32357cf319e50e9c8d8dab5ea876c72b0fd321b· acquired 2026-09-12 · releaseaa57f7510089d32357cf319e50e9c8d8dab5ea876c72b0fd321b· acquired 2026-09-12 · release8cda5f6854a1d32357cf319e50e9c8d8dab5ea876c72b0fd321b· acquired 2026-09-12 · releasead959cd691f5
- def compare_one(a, b):
Answer published by the source
- def add(x: int, y: int):
Answer published by the source
- def fib4(n: int):
Answer published by the source
- def encode_shift(s: str):
Answer published by the source
- from typing import List
Answer published by the source
- def rounded_avg(n, m):
Answer published by the source
- def minSubArraySum(nums):
Answer published by the source
- def Strongest_Extension(class_name, extensions):
Answer published by the source
- def change_base(x: int, base: int):
Answer published by the source
- def even_odd_palindrome(n):
Answer published by the source
- def sum_to_n(n: int):
Answer published by the source
- def get_row(lst, x):
Answer published by the source
- def car_race_collision(n: int):
Answer published by the source
- def unique(l: list):
Answer published by the source
- def is_sorted(lst):
Answer published by the source
- def order_by_points(nums):
Answer published by the source
- def find_max(words):
Answer published by the source
- def histogram(test):
Answer published by the source
- def file_name_check(file_name):
Answer published by the source
- def f(n):
Answer published by the source
- def sort_array(array):
Answer published by the source
- def select_words(s, n):
Answer published by the source
- def sort_even(l: list):
Answer published by the source
- def median(l: list):
Answer published by the source
- def is_equal_to_sum_even(n):
Answer published by the source
- from typing import List, Optional
Answer published by the source
- from typing import List
Answer published by the source
- def minPath(grid, k):
Answer published by the source
- def exchange(lst1, lst2):
Answer published by the source
- import math
Answer published by the source
- def strange_sort_list(lst):
Answer published by the source
- def largest_prime_factor(n: int):
Answer published by the source
- def intersection(interval1, interval2):
Answer published by the source
- import math
Answer published by the source
- def encode(message):
Answer published by the source
- def closest_integer(value):
Answer published by the source
- def string_sequence(n: int) -> str:
Answer published by the source
- def can_arrange(arr):
Answer published by the source
- def is_multiply_prime(a):
Answer published by the source
- def is_prime(n):
Answer published by the source
- def move_one_ball(arr):
Answer published by the source
- def incr_list(l: list):
Answer published by the source
- def unique_digits(x):
Answer published by the source
- def specialFilter(nums):
Answer published by the source
- def odd_count(lst):
Answer published by the source
- def fizz_buzz(n: int):
Answer published by the source
- def is_nested(string):
Answer published by the source
- from typing import List
Answer published by the source
- def check_if_last_char_is_a_letter(txt):
Answer published by the source
- def fibfib(n: int):
Answer published by the source