Discussion
0 posts
No posts in this thread yet.
benchmarks.wiki / Public workspace
MBPP
Code
Mostly Basic Python Problems asks for a short Python function from a one-sentence description, checked by assertions. The sanitized split is the hand-verified subset.
mbpp
Full text: eligible public problems include text, images and discussion.
History
Google researchers introduced Mostly Basic Python Problems in 2021 to study program synthesis from natural-language descriptions. The benchmark focused on short Python tasks suitable for checking with executable tests. The Google-hosted dataset recorded in this catalog was published on Hugging Face in March 2022, after the original research release.
How it was built
MBPP was assembled from crowd-sourced Python programming tasks, each accompanied by a natural-language description, reference code and assertion tests. The sanitized version is a hand-verified subset intended to improve task quality. This catalog imports 200 problems from the sanitized test selection and keeps both reference programs and assertions out of the displayed prompt. Candidate programs are judged by executing the tests rather than matching the reference text.
At a glance
What a problem looks like
A one-sentence task description (text in the full config, prompt in sanitized) with the assertion list that names the expected function. The reference code is kept out of the prompt.
How it is scored
The source ships a reference program with every problem; it stays in restricted storage and is never rendered. The assertion tests are part of the problem statement and are shown with it; correctness is decided by executing a candidate against the assertions, not by comparing it with the reference.
Metric: pass@1 on the assertion tests
Reported scores
Evaluation conditions differ between rows, so this is not a ranking. Rows marked vendor are reports by the model’s own organization.
| Model | Score | Metric | Setup | Reported by | Source |
|---|---|---|---|---|---|
| DeepSeek-V3DeepSeek | 75.4 | Pass@1 | Base model; 3-shot. | vendor (self-reported) | Base Model, Standard Benchmarks, MBPP (Pass@1), final column |
Why it is hard
The description is short enough to be ambiguous, so the model has to infer the intended signature and behaviour from the tests it cannot see.
Limitations and criticism
A short description can leave behavior or the expected function interface ambiguous, particularly when tests are withheld from the prompt. Passing a finite set of assertions does not establish correctness on untested inputs, and small standalone functions do not measure repository-scale software engineering. Comparisons also depend on whether they use the original or sanitized dataset and whether demonstrations or tests are supplied. Public availability creates a potential training-contamination route without establishing exposure for any particular model.
How much room is left
The vendor result recorded here reports DeepSeek-V3 at 75.4 pass@1 in a base-model, three-shot setup. That report leaves headroom under its stated conditions, but it does not identify the present ceiling of the benchmark. It should not be read as a result specifically on this catalog's 200 imported sanitized tasks unless the evaluation subset is independently confirmed.
Reference links
Cite
@article{austin2021program,
title={Program Synthesis with Large Language Models},
author={Austin, Jacob and Odena, Augustus and Nye, Maxwell and Bosma, Maarten and Michalewski, Henryk and Dohan, David and Jiang, Ellen and Cai, Carrie and Terry, Michael and Le, Quoc and others},
journal={arXiv preprint arXiv:2108.07732},
year={2021}Discussion
1 thread and 0 posts on this page. Anyone may reply; no account is required.
Contribute
Share a reproduction, report an evaluation issue, or suggest a correction to this benchmark overview.
Tasks
Whole public benchmark.
- Public problems
- 1,401
- Formats
- text (1,401)
- Problems with images
- 0
- Answer availability
- Answer published by the source (1,401)
- Configs
- full (974) · sanitized (427)
- Splits
- prompt (17) · test (757) · train (494) · validation (133)
- Awaiting a first discussion
- 1,401
- Problems with a discussion
- 0
- Public contributions
- 0
Provenance and terms
Pinned source revisions, newest first:
4bb6404fdc6cacfda99d4ac4205087b89d32030c· acquired 2026-09-23 · releasef1af1b73310c4bb6404fdc6cacfda99d4ac4205087b89d32030c· acquired 2026-09-11 · releasea9c6c87472554bb6404fdc6cacfda99d4ac4205087b89d32030c· acquired 2026-09-11 · releaseecd40ec0a69d4bb6404fdc6cacfda99d4ac4205087b89d32030c· acquired 2026-09-11 · release67cf16715b15
- Write a python function to check whether the two numbers differ at one bit…
Answer published by the source
- Write a function to find n’th smart number.
Answer published by the source
- Write a python function to toggle all even bits of a given number.
Answer published by the source
- Write a function to find the longest common subsequence for the given two…
Answer published by the source
- Write a python function to check whether the given number can be represented by…
Answer published by the source
- Write a python function to count numbers whose oth and nth bits are set.
Answer published by the source
- Write a function to extract maximum and minimum k elements in the given tuple.
Answer published by the source
- Write a python function to find the maximum sum of elements of list in a list of…
Answer published by the source
- Write a function to find the depth of a dictionary.
Answer published by the source
- Write a function that matches a word at the end of a string, with optional…
Answer published by the source
- Write a function to count sequences of given length having non-negative prefix…
Answer published by the source
- Write a function to find the smallest missing element in a sorted array.
Answer published by the source
- Write a function to generate a two-dimensional array.
Answer published by the source
- Write a function to find the list with maximum length using lambda function.
Answer published by the source
- Write a function to find frequency of the elements in a given list of lists…
Answer published by the source
- Write a function to compute the value of ncr%p.
Answer published by the source
- Write a function to create the next bigger number by rearranging the digits of a…
Answer published by the source
- Write a function to find the ration of positive numbers in an array of integers.
Answer published by the source
- Write a function to check whether the given month number contains 31 days or…
Answer published by the source
- Write a python function to count the upper case characters in a given string.
Answer published by the source
- Write a function to split the given string with multiple delimiters by using…
Answer published by the source
- Write a function to get a lucid number smaller than or equal to n.
Answer published by the source
- Write a function to find the item with maximum occurrences in a given list.
Answer published by the source
- Write a python function to check whether the given number can be represented by…
Answer published by the source
- Write a function to extract unique values from the given dictionary values.
Answer published by the source
- Write a function to find common index elements from three lists.
Answer published by the source
- Write a python function to find the minimum difference between any two elements…
Answer published by the source
- Write a python function to find the sum of fourth power of first n odd natural…
Answer published by the source
- Write a function to convert rgb color to hsv color.
Answer published by the source
- Write a function to find the sum of first even and odd number of a given list.
Answer published by the source
- Write a function to combine two given sorted lists using heapq module.
Answer published by the source
- Write a python function to left rotate the string.
Answer published by the source
- Write a function to find the nth jacobsthal-lucas number.
Answer published by the source
- Write a function to extract the even elements in the nested mixed tuple.
Answer published by the source
- Write a function to find words that are longer than n characters from a given…
Answer published by the source
- Write a function that returns integers x and y that satisfy ax + by = n as a…
Answer published by the source
- Write a function to find the sum of numbers in a list within a range specified…
Answer published by the source
- Write a python function that takes in an integer n and returns the sum of the…
Answer published by the source
- Write a function that checks whether a string contains the 'a' character…
Answer published by the source
- Write a function to find the third side of a right angled triangle.
Answer published by the source
- Write a function to convert a date of yyyy-mm-dd format to dd-mm-yyyy format.
Answer published by the source
- Write a function to remove odd characters in a string.
Answer published by the source
- Write a python function to check whether the given list contains consecutive…
Answer published by the source
- Write a function to remove lowercase substrings from a given string.
Answer published by the source
- Write a function to find whether all the given tuples have equal length or not.
Answer published by the source
- Write a function to check for the number of jumps required of given length to…
Answer published by the source
- Write a python function to count number of digits in a given string.
Answer published by the source
- Write a function to check whether the given number is a perfect square or not.…
Answer published by the source
- Write a function to find the sum of the largest contiguous sublist in the given…
Answer published by the source
- Write a python function to interchange the first and last elements in a list.
Answer published by the source