benchmarks.wiki / Public workspace
DROP
Reasoning
DROP is a reading-comprehension benchmark whose questions require discrete operations over the paragraph — addition, subtraction, comparison, counting, sorting. It is built so that extracting a span cannot answer them.
drop
Full text: eligible public problems include text, images and discussion.
Reference links
At a glance
What a problem looks like
A passage and a question about it. The answer spans and numbers are not part of the prompt.
How it is scored
The source ships answer spans with every question; this catalogue keeps them in restricted storage and never renders them.
Metric: exact match and macro-averaged F1 over the answer spans
Why it is hard
The passage contains the facts but not the answer. Getting it right means locating several numbers, deciding which operation relates them, and carrying it out exactly.
Cite
@article{dua2019drop,
title={DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs},
author={Dheeru Dua and Yizhong Wang and Pradeep Dasigi and Gabriel Stanovsky and Sameer Singh and Matt Gardner},
journal={arXiv preprint arXiv:1903.00161},
year={2019}
}Discussion
Contribute
Share a reproduction, report an evaluation issue, or suggest a correction to this benchmark overview.
Tasks
Whole public benchmark.
- Public problems
- 200
- Formats
- text (200)
- Problems with images
- 0
- Answer availability
- Answer published by the source (200)
- Configs
- default (200)
- Splits
- validation (200)
- Awaiting a first discussion
- 200
- Problems with a discussion
- 0
- Public contributions
- 0
Provenance and terms
Pinned source revisions, newest first:
95cda593fae71b60b5b19f82de3fcf3298c1239c· acquired 2026-09-12 · releasec2d8b15a0e8995cda593fae71b60b5b19f82de3fcf3298c1239c· acquired 2026-09-12 · releasefc596d0216e7
- How many in percent from the census weren't English?
Answer published by the source
- How many field goals were scored in the game?
Answer published by the source
- How many touchdowns did T.J. Houshmandzadeh have?
Answer published by the source
- How many yards was the longest passing touchdown?
Answer published by the source
- How many of the personnel were not non-commisioned officers?
Answer published by the source
- How many denominations of Pesos under 50 were there in 1944?
Answer published by the source
- How many field goals were 40 yards or over?
Answer published by the source
- How many in percent in the county from the census of 2000 weren't English?
Answer published by the source
- How many different denominations of centavos were there over 6?
Answer published by the source
- How many Korean immigrants were more men than woman and children?
Answer published by the source
- Which team won the game?
Answer published by the source
- How many field goals were there in the fourth quarter?
Answer published by the source
- How many total points were scored in the game?
Answer published by the source
- How many points did the Bears lead with by halftime?
Answer published by the source
- In the county according to the 2000 census, which group is smaller: people or…
Answer published by the source
- How many quarters did the Lions play before scoring any points?
Answer published by the source
- How many more active military personnel were in the Armed Forces than civil…
Answer published by the source
- How many yards was the longest field goal?
Answer published by the source
- How many years went by between the king giving a new fuero to the city and the…
Answer published by the source
- How many yards of field goals did Gould kick?
Answer published by the source
- Which group had more active military personnel, the military specialists or the…
Answer published by the source
- Where was the coronation that Charles V was supposed to attend?
Answer published by the source
- How many touchdowns were scored in the first quarter?
Answer published by the source
- What was more expensive, matches in 1944 or a kilogram of camote in 1945?
Answer published by the source
- How many new Peso notes were presented in 1944?
Answer published by the source
- How many yards longer was the longest field goal than the shortest?
Answer published by the source
- How many field goals of at least 40 yards were made?
Answer published by the source
- From what distance did Robbie Gould make more than one field goal attempt?
Answer published by the source
- When did the first battle against Guadalajara take place?
Answer published by the source
- Which group had less active military personnel, officers or soldiers?
Answer published by the source
- Which personnel were there more of in the Croatian Armed Forces, officers or…
Answer published by the source
- In the county according to the 2000 census, which group is larger: people or…
Answer published by the source
- Did Lockette play again?
Answer published by the source
- How many more Korean immigrant men were there that departed from the port of…
Answer published by the source
- How many more non commissioned officers than military specialists were in the…
Answer published by the source
- Which team scored the final TD of the game?
Answer published by the source
- How many years passed after Guadalajara was retaken and the Battle of Las Navas…
Answer published by the source
- In the county according to the 2000 census, which group is larger: people or…
Answer published by the source
- How many of the personnel were not civil servants?
Answer published by the source
- Which team allowed the least points in the first half?
Answer published by the source
- How many points did the Bears score in the second half?
Answer published by the source
- How many active military personnel and reserve are in the Croatian Armed Forces?
Answer published by the source
- How many points ahead were the Seahawks of the Cowboys?
Answer published by the source
- Where did the immigrants arrive first, Salina Cruz or Coatzacoalcos?
Answer published by the source
- Which player had the longest rushing TD?
Answer published by the source
- How many total points were scored?
Answer published by the source
- How many touchdowns did Crayton have?
Answer published by the source
- How many field goals did Shayne Graham kick in the first half?
Answer published by the source
- how many yards did Morris take off?
Answer published by the source
- Were there more men or women and children who departed from the port of…
Answer published by the source