Discussion
0 posts
No posts in this thread yet.
benchmarks.wiki / Public workspace
MBPP+
Code
MBPP+ takes the 378 problems of the sanitized MBPP set and multiplies their tests by roughly 35 times. It removes the same test-weakness inflation that HumanEval+ addresses.
mbpp-plus
Full text: eligible public problems include text, images and discussion.
History
The EvalPlus project developed MBPP+ to strengthen the execution-based evaluation of programs on MBPP. Its Hugging Face distribution was published in January 2024. The motivation was to catch incorrect programs that pass a small original test suite, a problem also addressed by HumanEval+.
How it was built
MBPP+ retains the 378 programming problems of the sanitized MBPP set and expands their tests by roughly 35 times. The task remains a short natural-language programming request, with candidate programs evaluated by running the expanded assertions. This catalogue imports 200 of the 378 problems and keeps reference programs and expanded assertions outside the displayed prompt.
At a glance
What a problem looks like
A short natural-language programming task. The reference program and the expanded assertions are not part of the prompt.
How it is scored
Correctness is decided by executing the candidate against the expanded assertions, so the reference program the source ships is not a graded answer; it stays in restricted storage.
Metric: pass@1 on the expanded tests
Why it is hard
Each task is a few lines of Python, but the specification is one sentence long, so most of the difficulty is in reading what is actually being asked for.
Limitations and criticism
Additional tests provide stronger evidence of correctness but cannot prove that a program satisfies a specification for every possible input. Short task descriptions can also leave edge cases underspecified, making the reference implementation and tests consequential parts of the evaluation. The benchmark covers small programming tasks rather than repository-scale development, and expanding tests does not remove potential exposure to the original public problems.
Reference links
Cite
@article{liu2023mbppplus,
title={Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation},
author={Jiawei Liu and Chunqiu Steven Xia and Yuyao Wang and Lingming Zhang},
journal={arXiv preprint arXiv:2305.01210},
year={2023}
}Discussion
1 thread and 0 posts on this page. Anyone may reply; no account is required.
Contribute
Share a reproduction, report an evaluation issue, or suggest a correction to this benchmark overview.
Tasks
Whole public benchmark.
- Public problems
- 378
- Formats
- code (378)
- Problems with images
- 0
- Answer availability
- Answer published by the source (378)
- Configs
- default (378)
- Splits
- test (378)
- Awaiting a first discussion
- 378
- Problems with a discussion
- 0
- Public contributions
- 0
Provenance and terms
Pinned source revisions, newest first:
b2d74c91837c3f2a20c1299ae98133cbe7cfa077· acquired 2026-09-12 · release10a7a2abecadb2d74c91837c3f2a20c1299ae98133cbe7cfa077· acquired 2026-09-12 · release532e203491c3b2d74c91837c3f2a20c1299ae98133cbe7cfa077· acquired 2026-09-12 · release147b31500dd9
- Write a function to convert a snake case string to camel case string.
Answer published by the source
- Write a function to sort each sublist of strings in a given list of lists.
Answer published by the source
- Write a function to find the maximum absolute product between numbers in pairs…
Answer published by the source
- Write a function to sum all amicable numbers from 1 to a specified number.
Answer published by the source
- Write a python function to find the maximum difference between any two elements…
Answer published by the source
- Write a function to remove all elements from a given list present in another…
Answer published by the source
- Write a function that counts the number of pairs of integers in a list that xor…
Answer published by the source
- Write a function to find the perimeter of a regular pentagon from the length of…
Answer published by the source
- Write a function that takes in a string and character, replaces blank spaces in…
Answer published by the source
- Write a python function that returns the number of integer elements in a given…
Answer published by the source
- Write a function to maximize the given two tuples.
Answer published by the source
- Write a function that takes in a list and an integer L and splits the given list…
Answer published by the source
- Write a function to find the largest sum of a contiguous array in the modified…
Answer published by the source
- Write a function to flatten a given nested list structure.
Answer published by the source
- Write a function to find the count of all binary sequences of length 2n such…
Answer published by the source
- Write a function to apply a given format string to all of the elements in a…
Answer published by the source
- Write a function to check if given tuple contains no duplicates.
Answer published by the source
- Write a python function to find whether the parity of a given number is odd.
Answer published by the source
- Write a python function to find the element of a list having maximum length.
Answer published by the source
- Write a function to round every number of a given list of numbers and print the…
Answer published by the source
- Write a function to check whether the entered number is greater than the…
Answer published by the source
- Write a function to find the first adverb and their positions in a given…
Answer published by the source
- Write a function to find cubes of individual elements in a list.
Answer published by the source
- Write a function to extract specified size of strings from a given list of…
Answer published by the source
- Write a function to reverse each string in a given list of string values.
Answer published by the source
- Write a function to remove uppercase substrings from a given string.
Answer published by the source
- Write a python function to get the first element of each sublist.
Answer published by the source
- Write a function to find all possible combinations of the elements of a given…
Answer published by the source
- Write a python function which takes a list of integers and only returns the odd…
Answer published by the source
- Write a python function to split a string into characters.
Answer published by the source
- Write a function to sort each sublist of strings in a given list of lists.
Answer published by the source
- Write a python function to find the largest number that can be formed with the…
Answer published by the source
- Write a function to find the nth octagonal number.
Answer published by the source
- Write a function to get all lucid numbers smaller than or equal to a given…
Answer published by the source
- Write a python function to find nth bell number.
Answer published by the source
- Write a function to find the cumulative sum of all the values that are present…
Answer published by the source
- The input is defined as two lists of the same length. Write a function to count…
Answer published by the source
- Write a function that takes base and power as arguments and calculate the sum of…
Answer published by the source
- Write a python function to toggle bits of the number except the first and the…
Answer published by the source
- Write a function to check whether it follows the sequence given in the patterns…
Answer published by the source
- Write a function to find numbers within a given range from startnum ti endnum…
Answer published by the source
- Write a function to check if the given array represents min heap or not.…
Answer published by the source
- Write a function to remove all the words with k length in the given string.
Answer published by the source
- Write a python function to check whether every even index contains even numbers…
Answer published by the source
- Write a function to remove tuples from the given tuple.
Answer published by the source
- Write a python function to find the sum of an array.
Answer published by the source
- Write a function to that rotate left bits by d bits a given number. We assume…
Answer published by the source
- Write a function to find the nth tetrahedral number.
Answer published by the source
- Write a python function to find the first odd number in a given list of numbers.
Answer published by the source
- Write a function to check if the given tuples contain the k or not.
Answer published by the source