benchmarks.wiki

benchmarks.wiki / Public workspace

DROP

Reasoning

DROP is a reading-comprehension benchmark whose questions require discrete operations over the paragraph — addition, subtraction, comparison, counting, sorting. It is built so that extracting a span cannot answer them.

drop

Full text9,535 tasks

Official source

Full text: eligible public problems include text, images and discussion.

Download answersBrowse the problemsAnswersOpen a random oneProblems awaiting a first attempt

History

The Allen Institute for AI and collaborating researchers introduced DROP in 2019 to test reading comprehension that requires discrete reasoning over a passage. It targeted operations such as counting, comparison and arithmetic that were poorly captured by benchmarks focused on locating an answer span. The March 2022 publication date for this source describes the ucinlp Hugging Face distribution rather than the original benchmark release.

How it was built

DROP was built from Wikipedia passages, with crowdworkers writing questions that require reasoning over the information they contain. Its collection process used an existing question-answering model as an adversary to discourage questions that the model could already answer. Answers can be numbers, dates or text spans, and the source validation split contains 9,535 questions. Evaluation uses exact match and an answer-level F1 measure.

At a glance

Metric
exact match and macro-averaged F1 over the answer spans
Size
9,535 problems imported from the default / validation selection (the full validation split).
License
cc-by-sa-4.0 (declared on the Hugging Face dataset card)
Released
Dataset published on Hugging Face 2022-03
Maintainer
ucinlp (source organisation)
Answer access
Columns outside the prompt allowlist, including any answer or reference solution, are separated into restricted storage and never rendered.
Highest recorded
89 F1 — DeepSeek-V3 (vendor, self-reported)

What a problem looks like

A passage and a question about it. The answer spans and numbers are not part of the prompt.

Read an actual problem

How it is scored

The source ships answer spans with every question; this catalogue keeps them in restricted storage and never renders them.

Metric: exact match and macro-averaged F1 over the answer spans

Reported scores

Evaluation conditions differ between rows, so this is not a ranking. Rows marked vendor are reports by the model’s own organization.

ModelScoreMetricSetupReported bySource
DeepSeek-V3DeepSeek89F1Base model; 3-shot.vendor (self-reported)Base Model, Standard Benchmarks, DROP (F1), final column

Why it is hard

The passage contains the facts but not the answer. Getting it right means locating several numbers, deciding which operation relates them, and carrying it out exactly.

Limitations and criticism

DROP combines language understanding with discrete operations, so an incorrect answer does not isolate whether the failure came from reading, selecting facts or calculation. It includes span answers as well as numerical answers; it is not a pure arithmetic test. Exact match and F1 grade the final answer rather than the derivation, and partial overlap credit should not be read as proof of a partially correct reasoning process.

How much room is left

The vendor report recorded here gives the DeepSeek-V3 base model an F1 score of 89.0 in a three-shot setup. This is high on the reported metric but still below its ceiling, and it does not imply that 89 percent of questions were answered exactly correctly. That single result is insufficient to establish current saturation or the performance of systems using other prompting and evaluation settings.

Cite

@article{dua2019drop,
  title={DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs},
  author={Dheeru Dua and Yizhong Wang and Pradeep Dasigi and Gabriel Stanovsky and Sameer Singh and Matt Gardner},
  journal={arXiv preprint arXiv:1903.00161},
  year={2019}
}

Discussion

1 thread and 0 posts on this page. Anyone may reply; no account is required.

Contribute

Share a reproduction, report an evaluation issue, or suggest a correction to this benchmark overview.

Tasks

Whole public benchmark.

Public problems
9,535
Formats
text (9,535)
Problems with images
0
Answer availability
Answer published by the source (9,535)
Configs
default (9,535)
Splits
validation (9,535)
Awaiting a first discussion
9,535
Problems with a discussion
0
Public contributions
0

Provenance and terms

Pinned source revisions, newest first:

  • 95cda593fae71b60b5b19f82de3fcf3298c1239c · acquired 2026-09-12 · release c2d8b15a0e89
  • 95cda593fae71b60b5b19f82de3fcf3298c1239c · acquired 2026-09-12 · release 974819cfe356
  • 95cda593fae71b60b5b19f82de3fcf3298c1239c · acquired 2026-09-12 · release fc596d0216e7
Licence
Not recorded. This catalogue has not verified redistribution terms.
Redistribution terms
Not recorded. This catalogue has not verified redistribution terms.
Publisher requests
Not recorded. This catalogue has not verified redistribution terms.
Terms last checked
Never