benchmarks.wiki

benchmarks.wiki / Public workspace

DS-1000

Code / Science

DS-1000 builds 1,000 data-science problems from real StackOverflow questions across seven libraries including NumPy, Pandas and Matplotlib. The problems are perturbed so that recalling the original answer does not solve them.

ds-1000

Full text200 tasksProblems imported

Official source

Full text: eligible public problems include text, images and discussion.

Browse the problemsOpen a random oneProblems awaiting a first attempt

At a glance

Metric
accuracy (the source paper reports the best public system at 43.3%)
Size
200 problems imported from the default / test selection of 1,000, with a source-row limit of 200.
License
cc-by-sa-4.0 (declared on the Hugging Face dataset card)
Released
Dataset published on Hugging Face 2024-04
Maintainer
xlangai (source organisation)
Answer access
Columns outside the prompt allowlist, including any answer or reference solution, are separated into restricted storage and never rendered.

What a problem looks like

A data-science problem with the surrounding code context stated in the prompt. The reference code and the checking context are not part of the prompt.

Read an actual problem

How it is scored

Correctness is decided by executing the candidate inside the problem’s own code context, so the reference code the source ships is not a graded answer; it stays in restricted storage.

Metric: accuracy (the source paper reports the best public system at 43.3%)

Why it is hard

The problems come from people who were actually stuck, and the perturbations mean the remembered StackOverflow answer is usually close but wrong.

Cite

@article{lai2022ds1000,
  title={DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation},
  author={Yuhang Lai and Chengxi Li and Yiming Wang and Tianyi Zhang and Ruiqi Zhong and Luke Zettlemoyer and Scott Wen-tau Yih and Daniel Fried and Sida Wang and Tao Yu},
  journal={arXiv preprint arXiv:2211.11501},
  year={2022}
}

Discussion

Contribute

Share a reproduction, report an evaluation issue, or suggest a correction to this benchmark overview.

Tasks

Whole public benchmark.

Public problems
200
Formats
code (200)
Problems with images
0
Answer availability
Answer published by the source (200)
Configs
default (200)
Splits
test (200)
Awaiting a first discussion
200
Problems with a discussion
0
Public contributions
0

Provenance and terms

Pinned source revisions, newest first:

  • 4416080ac5cb80bdf7576aefb8f9a0b4d5426a44 · acquired 2026-09-12 · release 4346a40c861b
  • 4416080ac5cb80bdf7576aefb8f9a0b4d5426a44 · acquired 2026-09-12 · release b578ca080b3c
Licence
Not recorded. This catalogue has not verified redistribution terms.
Redistribution terms
Not recorded. This catalogue has not verified redistribution terms.
Publisher requests
Not recorded. This catalogue has not verified redistribution terms.
Terms last checked
Never