benchmarks.wiki / Public workspace
MBPP+
Code
MBPP+ takes the 378 problems of the sanitized MBPP set and multiplies their tests by roughly 35 times. It removes the same test-weakness inflation that HumanEval+ addresses.
mbpp-plus
Full text: eligible public problems include text, images and discussion.
Reference links
At a glance
What a problem looks like
A short natural-language programming task. The reference program and the expanded assertions are not part of the prompt.
How it is scored
Correctness is decided by executing the candidate against the expanded assertions, so the reference program the source ships is not a graded answer; it stays in restricted storage.
Metric: pass@1 on the expanded tests
Why it is hard
Each task is a few lines of Python, but the specification is one sentence long, so most of the difficulty is in reading what is actually being asked for.
Cite
@article{liu2023mbppplus,
title={Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation},
author={Jiawei Liu and Chunqiu Steven Xia and Yuyao Wang and Lingming Zhang},
journal={arXiv preprint arXiv:2305.01210},
year={2023}
}Discussion
Contribute
Share a reproduction, report an evaluation issue, or suggest a correction to this benchmark overview.
Tasks
Whole public benchmark.
- Public problems
- 200
- Formats
- code (200)
- Problems with images
- 0
- Answer availability
- Answer published by the source (200)
- Configs
- default (200)
- Splits
- test (200)
- Awaiting a first discussion
- 200
- Problems with a discussion
- 0
- Public contributions
- 0
Provenance and terms
Pinned source revisions, newest first:
b2d74c91837c3f2a20c1299ae98133cbe7cfa077· acquired 2026-09-12 · release10a7a2abecadb2d74c91837c3f2a20c1299ae98133cbe7cfa077· acquired 2026-09-12 · release532e203491c3
- Write a function to convert a snake case string to camel case string.
Answer published by the source
- Write a function to find the Eulerian number a(n, m).
Answer published by the source
- Write a function to sort each sublist of strings in a given list of lists.
Answer published by the source
- Write a python function to count true booleans in the given list.
Answer published by the source
- Write a function to find the maximum absolute product between numbers in pairs…
Answer published by the source
- Write a function to sum all amicable numbers from 1 to a specified number.
Answer published by the source
- Write a function to convert a tuple to a string.
Answer published by the source
- Write a function to sort a list of elements.
Answer published by the source
- Write a python function to find the maximum difference between any two elements…
Answer published by the source
- Write a function to remove all elements from a given list present in another…
Answer published by the source
- Write a function that counts the number of pairs of integers in a list that xor…
Answer published by the source
- Write a function to find the perimeter of a regular pentagon from the length of…
Answer published by the source
- Write a function that takes in a string and character, replaces blank spaces in…
Answer published by the source
- Write a python function that returns the number of integer elements in a given…
Answer published by the source
- Write a python function that takes in a non-negative number and returns the…
Answer published by the source
- Write a function to maximize the given two tuples.
Answer published by the source
- Write a function that takes in a list and an integer L and splits the given list…
Answer published by the source
- Write a function that takes in a list and an integer n and splits a list for…
Answer published by the source
- Write a function to find the ascii value of a character.
Answer published by the source
- Write a function to find the nth decagonal number.
Answer published by the source
- Write a python function to check if the elements of a given list are unique or…
Answer published by the source
- Write a function that takes in a list and element and checks whether all items…
Answer published by the source
- Write a function to find the largest sum of a contiguous array in the modified…
Answer published by the source
- Write a python function takes in an integer n and returns the sum of squares of…
Answer published by the source
- Write a python function to find quotient of two numbers (rounded down to the…
Answer published by the source
- Write a function to flatten a given nested list structure.
Answer published by the source
- Write a function to find the count of all binary sequences of length 2n such…
Answer published by the source
- Write a function to find the depth of a dictionary.
Answer published by the source
- Write a function to find the maximum sum of increasing subsequence from prefix…
Answer published by the source
- Write a python function to set the left most unset bit.
Answer published by the source
- Write a python function to find the highest power of 2 that is less than or…
Answer published by the source
- Write a function to apply a given format string to all of the elements in a…
Answer published by the source
- Write a function to convert more than one list to nested dictionary.
Answer published by the source
- Write a function to check if given tuple contains no duplicates.
Answer published by the source
- Write a python function to find whether the parity of a given number is odd.
Answer published by the source
- Write a python function to find the element of a list having maximum length.
Answer published by the source
- Write a function to round every number of a given list of numbers and print the…
Answer published by the source
- Write a function to find the median length of a trapezium.
Answer published by the source
- Write a function to check whether the entered number is greater than the…
Answer published by the source
- Write a python function to find the largest number that can be formed with the…
Answer published by the source
- Write a function to find the nth octagonal number.
Answer published by the source
- Write a function to sort a list of tuples using the second value of each tuple.
Answer published by the source
- Write a python function to count the number of positive numbers in a list.
Answer published by the source
- Write a python function to check whether the given array is monotonic or not.
Answer published by the source
- Write a function to find all words which are at least 4 characters long in a…
Answer published by the source
- Write a function to check whether it follows the sequence given in the patterns…
Answer published by the source
- Write a function to find squares of individual elements in a list.
Answer published by the source
- Write a function to find the nth tetrahedral number.
Answer published by the source
- Write a python function to find the length of the longest word.
Answer published by the source
- Write a function to multiply all the numbers in a list and divide with the…
Answer published by the source