HumanEval+

Reference links

Repository

• Official repository

Issues

• Dataset discussions

Paper

• arXiv:2305.01210

Dataset

• Source dataset

At a glance

Metric

pass@1 on the expanded tests

Size

All 164 problems imported from the default / test selection, with a source-row limit of 200.

License

apache-2.0 (declared on the Hugging Face dataset card)

Released

Dataset published on Hugging Face 2024-01

Maintainer

evalplus (source organisation)

Answer access

Columns outside the prompt allowlist, including any answer or reference solution, are separated into restricted storage and never rendered.

What a problem looks like

A Python function signature with its docstring, plus the entry-point name. The reference implementation and the test suite are not part of the prompt.

Read an actual problem

How it is scored

Correctness is decided by executing the candidate against the expanded test suite, so the reference implementation the source ships is not a graded answer; it stays in restricted storage.

Metric: pass@1 on the expanded tests

Why it is hard

The problems are short, but the added tests probe the edge cases the original suite missed, so a solution that only handles the obvious path now fails.

Cite

@article{liu2023humanevalplus,
  title={Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation},
  author={Jiawei Liu and Chunqiu Steven Xia and Yuyao Wang and Lingming Zhang},
  journal={arXiv preprint arXiv:2305.01210},
  year={2023}
}

Related benchmarks

Discussion

Contribute

Share a reproduction, report an evaluation issue, or suggest a correction to this benchmark overview.

• Discussion

Tasks

Whole public benchmark.

Public problems

164

Formats

code (164)

Problems with images

0

Answer availability

Answer published by the source (164)

Configs

default (164)

Splits

test (164)

Awaiting a first discussion

164

Problems with a discussion

0

Public contributions

0

• def compare_one(a, b):

  Answer published by the source

• def add(x: int, y: int):

  Answer published by the source

• def fib4(n: int):

  Answer published by the source

• def encode_shift(s: str):

  Answer published by the source

• from typing import List

  Answer published by the source

• def rounded_avg(n, m):

  Answer published by the source

• def minSubArraySum(nums):

  Answer published by the source

• def Strongest_Extension(class_name, extensions):

  Answer published by the source

• def change_base(x: int, base: int):

  Answer published by the source

• def even_odd_palindrome(n):

  Answer published by the source

• def sum_to_n(n: int):

  Answer published by the source

• def get_row(lst, x):

  Answer published by the source

• def car_race_collision(n: int):

  Answer published by the source

• def unique(l: list):

  Answer published by the source

• def is_sorted(lst):

  Answer published by the source

• def order_by_points(nums):

  Answer published by the source

• def find_max(words):

  Answer published by the source

• def histogram(test):

  Answer published by the source

• def file_name_check(file_name):

  Answer published by the source

• def f(n):

  Answer published by the source

• def sort_array(array):

  Answer published by the source

• def select_words(s, n):

  Answer published by the source

• def sort_even(l: list):

  Answer published by the source

• def median(l: list):

  Answer published by the source

• def is_equal_to_sum_even(n):

  Answer published by the source

• from typing import List, Optional

  Answer published by the source

• from typing import List

  Answer published by the source

• def minPath(grid, k):

  Answer published by the source

• def exchange(lst1, lst2):

  Answer published by the source

• import math

  Answer published by the source

• def strange_sort_list(lst):

  Answer published by the source

• def largest_prime_factor(n: int):

  Answer published by the source

• def intersection(interval1, interval2):

  Answer published by the source

• import math

  Answer published by the source

• def encode(message):

  Answer published by the source

• def closest_integer(value):

  Answer published by the source

• def string_sequence(n: int) -> str:

  Answer published by the source

• def can_arrange(arr):

  Answer published by the source

• def is_multiply_prime(a):

  Answer published by the source

• def is_prime(n):

  Answer published by the source

• def move_one_ball(arr):

  Answer published by the source

• def incr_list(l: list):

  Answer published by the source

• def unique_digits(x):

  Answer published by the source

• def specialFilter(nums):

  Answer published by the source

• def odd_count(lst):

  Answer published by the source

• def fizz_buzz(n: int):

  Answer published by the source

• def is_nested(string):

  Answer published by the source

• from typing import List

  Answer published by the source

• def check_if_last_char_is_a_letter(txt):

  Answer published by the source

• def fibfib(n: int):

  Answer published by the source