benchmarks.wiki

benchmarks.wiki / Public workspace

DS-1000

Code / Science

DS-1000 builds 1,000 data-science problems from real StackOverflow questions across seven libraries including NumPy, Pandas and Matplotlib. The problems are perturbed so that recalling the original answer does not solve them.

ds-1000

Full text1,000 tasks

Official source

Full text: eligible public problems include text, images and discussion.

Download answersBrowse the problemsAnswersOpen a random oneProblems awaiting a first attempt

How it was built

DS-1000 contains 1,000 data-science programming problems derived from real Stack Overflow questions across seven libraries, including NumPy, pandas and Matplotlib. The construction process includes perturbing the source problems so that reproducing an original posted answer is not sufficient for the modified task. Candidate code is evaluated by execution within the problem's checking context, rather than by textual comparison with reference code. The catalog imports 200 problems from the full test selection.

At a glance

Metric
accuracy (the source paper reports the best public system at 43.3%)
Size
1,000 problems imported from the default / test selection (the full split).
License
cc-by-sa-4.0 (declared on the Hugging Face dataset card)
Released
Dataset published on Hugging Face 2024-04
Maintainer
xlangai (source organisation)
Answer access
Columns outside the prompt allowlist, including any answer or reference solution, are separated into restricted storage and never rendered.

What a problem looks like

A data-science problem with the surrounding code context stated in the prompt. The reference code and the checking context are not part of the prompt.

Read an actual problem

How it is scored

Correctness is decided by executing the candidate inside the problem’s own code context, so the reference code the source ships is not a graded answer; it stays in restricted storage.

Metric: accuracy (the source paper reports the best public system at 43.3%)

Why it is hard

The problems come from people who were actually stuck, and the perturbations mean the remembered StackOverflow answer is usually close but wrong.

Limitations and criticism

The benchmark measures bounded programming tasks in a specific set of data-science libraries, rather than an entire analysis workflow or the scientific validity of an investigation. Perturbing public questions reduces the usefulness of verbatim recall but does not establish that the source material or benchmark itself was absent from training. Execution results also depend on the library environment, so comparisons require compatible dependencies and evaluation code.

How much room is left

This catalogue's metric record cites the source paper's best public system at 43.3% accuracy, which leaves substantial headroom in that evaluation. That is a historical result rather than a measurement of the systems available today, and no leaderboard entry is recorded here. It supports the benchmark's original difficulty without establishing its current distance from saturation.

Cite

@article{lai2022ds1000,
  title={DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation},
  author={Yuhang Lai and Chengxi Li and Yiming Wang and Tianyi Zhang and Ruiqi Zhong and Luke Zettlemoyer and Scott Wen-tau Yih and Daniel Fried and Sida Wang and Tao Yu},
  journal={arXiv preprint arXiv:2211.11501},
  year={2022}
}

Discussion

1 thread and 0 posts on this page. Anyone may reply; no account is required.

Contribute

Share a reproduction, report an evaluation issue, or suggest a correction to this benchmark overview.

Tasks

Whole public benchmark.

Public problems
1,000
Formats
code (1,000)
Problems with images
0
Answer availability
Answer published by the source (1,000)
Configs
default (1,000)
Splits
test (1,000)
Awaiting a first discussion
1,000
Problems with a discussion
0
Public contributions
0

Provenance and terms

Pinned source revisions, newest first:

  • 4416080ac5cb80bdf7576aefb8f9a0b4d5426a44 · acquired 2026-09-12 · release b578ca080b3c
  • 4416080ac5cb80bdf7576aefb8f9a0b4d5426a44 · acquired 2026-09-12 · release 8a121e02617b
  • 4416080ac5cb80bdf7576aefb8f9a0b4d5426a44 · acquired 2026-09-12 · release 4346a40c861b
Licence
Not recorded. This catalogue has not verified redistribution terms.
Redistribution terms
Not recorded. This catalogue has not verified redistribution terms.
Publisher requests
Not recorded. This catalogue has not verified redistribution terms.
Terms last checked
Never