benchmarks.wiki / Public workspace
MBPP
Code
Mostly Basic Python Problems asks for a short Python function from a one-sentence description, checked by assertions. The sanitized split is the hand-verified subset.
mbpp
Full text: eligible public problems include text, images and discussion.
Reference links
At a glance
What a problem looks like
A one-sentence task description in prompt. The reference code and the assertion list are kept out of the prompt.
How it is scored
For the 200 imported problems, every column outside the prompt is separated into restricted storage, but the converter does not interpret it, so this catalogue does not claim a usable answer key exists.
Why it is hard
The description is short enough to be ambiguous, so the model has to infer the intended signature and behaviour from the tests it cannot see.
Cite
@article{austin2021program,
title={Program Synthesis with Large Language Models},
author={Austin, Jacob and Odena, Augustus and Nye, Maxwell and Bosma, Maarten and Michalewski, Henryk and Dohan, David and Jiang, Ellen and Cai, Carrie and Terry, Michael and Le, Quoc and others},
journal={arXiv preprint arXiv:2108.07732},
year={2021}Discussion
Contribute
Share a reproduction, report an evaluation issue, or suggest a correction to this benchmark overview.
Tasks
Whole public benchmark.
- Public problems
- 200
- Formats
- text (200)
- Problems with images
- 0
- Answer availability
- Answer availability not recorded (200)
- Configs
- sanitized (200)
- Splits
- test (200)
- Awaiting a first discussion
- 200
- Problems with a discussion
- 0
- Public contributions
- 0
Provenance and terms
Pinned source revisions, newest first:
4bb6404fdc6cacfda99d4ac4205087b89d32030c· acquired 2026-09-11 · releasea9c6c8747255
- Write a function to convert a snake case string to camel case string.
Answer availability not recorded
- Write a python function to count true booleans in the given list.
Answer availability not recorded
- Write a function to check whether all dictionaries in a list are empty or not.
Answer availability not recorded
- Write a function to convert all possible convertible elements in a list of lists…
Answer availability not recorded
- Write a function to multiply two integers.
Answer availability not recorded
- Write a function to find words that are longer than n characters from a given…
Answer availability not recorded
- Write a python function to reverse only the vowels of a given string (where y is…
Answer availability not recorded
- Write a function that returns integers x and y that satisfy ax + by = n as a…
Answer availability not recorded
- Write a function to find the sum of numbers in a list within a range specified…
Answer availability not recorded
- Write a function to remove characters from the first string which are present in…
Answer availability not recorded
- Write a python function to remove the characters which have odd index values of…
Answer availability not recorded
- Write a function that takes an array and finds the maximum sum of a bitonic…
Answer availability not recorded
- Write a python function that takes in a tuple and an element and counts the…
Answer availability not recorded
- Write a function that takes in a list and length n, and generates all…
Answer availability not recorded
- Write a function to find the nth newman–shanks–williams prime number.
Answer availability not recorded
- Write a function that takes in two tuples and performs mathematical division…
Answer availability not recorded
- Write a function that takes in a list and an integer L and splits the given list…
Answer availability not recorded
- Write a function to calculate a dog's age in dog's years.
Answer availability not recorded
- Write a function to find the lateral surface area of a cube given its side…
Answer availability not recorded
- Write a python function that takes in an integer n and returns the sum of the…
Answer availability not recorded
- Write a python function to find the sum of even numbers at even positions of a…
Answer availability not recorded
- Write a python function that takes in an integer n and finds the sum of the…
Answer availability not recorded
- Write a python function that takes in a positive integer n and finds the sum of…
Answer availability not recorded
- Write a function that takes in the radius and height of a cylinder and returns…
Answer availability not recorded
- Write a python function to check if the elements of a given list are unique or…
Answer availability not recorded
- Write a function that checks whether a string contains the 'a' character…
Answer availability not recorded
- Write a function to find the largest sum of a contiguous array in the modified…
Answer availability not recorded
- Write a function to find the third side of a right angled triangle.
Answer availability not recorded
- Write a function to find the depth of a dictionary.
Answer availability not recorded
- Write a python function to find element at a given index after number of…
Answer availability not recorded
- Write a function to return two words from a list of words starting with letter…
Answer availability not recorded
- Write a python function to set the left most unset bit.
Answer availability not recorded
- Write a function to find the volume of a cone.
Answer availability not recorded
- Write a python function to find the highest power of 2 that is less than or…
Answer availability not recorded
- Write a function to convert more than one list to nested dictionary.
Answer availability not recorded
- Write a function to check whether the given string starts and ends with the same…
Answer availability not recorded
- Write a python function to find whether the parity of a given number is odd.
Answer availability not recorded
- Write a function to create the next bigger number by rearranging the digits of a…
Answer availability not recorded
- Write a function to find the minimum value in a given heterogeneous list.
Answer availability not recorded
- Write a function to find common first element in given list of tuple.
Answer availability not recorded
- Write a function to find the nth octagonal number.
Answer availability not recorded
- Write a function to find the maximum difference between available pairs in the…
Answer availability not recorded
- Write a function to find the number of ways to partition a set of Bell numbers.
Answer availability not recorded
- Write a function to check whether a list contains the given sublist or not.
Answer availability not recorded
- Write a function to find whether all the given tuples have equal length or not.
Answer availability not recorded
- Write a python function to check whether the length of the word is odd or not.
Answer availability not recorded
- Write a function to find nth centered hexagonal number.
Answer availability not recorded
- Write a function to get the frequency of all the elements in a list, returned as…
Answer availability not recorded
- Write a function to check whether the given number is undulating or not.
Answer availability not recorded
- Write a function to calculate the value of 'a' to the power 'b'.
Answer availability not recorded