benchmarks.wiki / Public workspace
BoolQ
Reasoning
BoolQ pairs naturally occurring yes/no questions from search queries with a passage that decides them. The format is simple, but the inference it needs is consistently heavier than it looks.
boolq
Full text: eligible public problems include text, images and discussion.
Reference links
At a glance
What a problem looks like
A yes/no question and the passage it is asked against. The boolean answer is not part of the prompt.
How it is scored
The source ships the boolean answer with every question; this catalogue keeps it in restricted storage and never renders it.
Metric: accuracy over the 3,270 validation questions
Why it is hard
The questions were written by people who wanted to know the answer, not to make a test, so they rarely line up with a single sentence of the passage.
Cite
@article{clark2019boolq,
title={BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions},
author={Christopher Clark and Kenton Lee and Ming-Wei Chang and Tom Kwiatkowski and Michael Collins and Kristina Toutanova},
journal={arXiv preprint arXiv:1905.10044},
year={2019}
}Discussion
Contribute
Share a reproduction, report an evaluation issue, or suggest a correction to this benchmark overview.
Tasks
Whole public benchmark.
- Public problems
- 200
- Formats
- text (200)
- Problems with images
- 0
- Answer availability
- Answer published by the source (200)
- Configs
- default (200)
- Splits
- validation (200)
- Awaiting a first discussion
- 200
- Problems with a discussion
- 0
- Public contributions
- 0
Provenance and terms
Pinned source revisions, newest first:
35b264d03638db9f4ce671b711558bf7ff0f80d5· acquired 2026-09-12 · releasefc6c571be9dd35b264d03638db9f4ce671b711558bf7ff0f80d5· acquired 2026-09-12 · release9192c36c5220
- was eye of the tiger written for rocky
Answer published by the source
- did to kill a mockingbird win an academy award
Answer published by the source
- is there such a thing as a floating island
Answer published by the source
- is ncis new orleans over for the season
Answer published by the source
- is columbus day a national holiday in the united states
Answer published by the source
- can root 2 be written as a fraction
Answer published by the source
- are all the members of the band still alive
Answer published by the source
- are the marvel tv shows in the mcu
Answer published by the source
- does p o box come before street address
Answer published by the source
- can you sell a gun privately in florida
Answer published by the source
- do female ferrets die if they don't mate
Answer published by the source
- can carbon form polar covalent bonds with hydrogen
Answer published by the source
- is there an official release date for the last airbender 2
Answer published by the source
- does the president live in the white house
Answer published by the source
- is there a difference between hydroxyzine hcl and hydroxyzine pam
Answer published by the source
- is mount everest a part of the himalayas
Answer published by the source
- do blue and pink cotton candy taste the same
Answer published by the source
- can one travel faster than the speed of light
Answer published by the source
- is hindi is our national language of india
Answer published by the source
- is the united states a part of the european union
Answer published by the source
- has anyone hit a hole in one on a par 5
Answer published by the source
- can you get the death penalty as a minor
Answer published by the source
- are there actual bodies in the tomb of the unknown soldier
Answer published by the source
- is there a military base in san diego
Answer published by the source
- is a tablespoon bigger than a dessert spoon
Answer published by the source
- does only the winner get money on tipping point
Answer published by the source
- is pureed tomatoes the same as tomato sauce
Answer published by the source
- is this the last season of bunk'd
Answer published by the source
- was there a draft in the revolutionary war
Answer published by the source
- is a pickle the same as a gherkin
Answer published by the source
- have the capitals ever win the stanley cup
Answer published by the source
- is ivory coast and cote d ivoire the same
Answer published by the source
- can you cut the stock off a shotgun
Answer published by the source
- is all of new zealand in the same time zone
Answer published by the source
- was the sears tower the tallest building in the world
Answer published by the source
- can an african team win the world cup
Answer published by the source
- will there be a season 3 of here come the habibs
Answer published by the source
- is scar alive in the lion king 2
Answer published by the source
- is the other boleyn girl part of a series
Answer published by the source
- did ireland qualify for the 2018 world cup
Answer published by the source
- did boo the world's cutest dog die
Answer published by the source
- is pain experienced in a missing body part or paralyzed area
Answer published by the source
- does damon and elena get together in season 3
Answer published by the source
- does deadpool have a kid in the comics
Answer published by the source
- is antarctica the same as the south pole
Answer published by the source
- can a tree have more than one trunk
Answer published by the source
- do competitive inhibitors bind to the active site
Answer published by the source
- do they still have fox hunts in england
Answer published by the source
- is there a word with q without u
Answer published by the source
- is fate and the furious the last movie
Answer published by the source