benchmarks.wiki

benchmarks.wiki / Public workspace

BoolQ

Reasoning

BoolQ pairs naturally occurring yes/no questions from search queries with a passage that decides them. The format is simple, but the inference it needs is consistently heavier than it looks.

boolq

Full text3,270 tasks

Official source

Full text: eligible public problems include text, images and discussion.

Download answersBrowse the problemsAnswersOpen a random oneProblems awaiting a first attempt

History

Google researchers introduced BoolQ in 2019 to study yes-or-no questions that arise naturally, rather than questions written specifically for a reading-comprehension test. The motivation was to test whether systems could infer an answer from a passage when the question did not simply repeat its wording. The March 2022 Hugging Face publication date recorded for this source refers to the hosted dataset, not the original research release.

How it was built

BoolQ pairs naturally occurring questions from search queries with passages that provide evidence for a boolean answer. Human annotation establishes whether the passage supports a yes or no response. The source validation split contains 3,270 questions, of which this catalogue imports 200; accuracy over the full validation split is distinct from accuracy over that smaller selection.

At a glance

Metric
accuracy over the 3,270 validation questions
Size
3,270 problems imported from the default / validation selection (the full validation split).
License
cc-by-sa-3.0 (declared on the Hugging Face dataset card)
Released
Dataset published on Hugging Face 2022-03
Maintainer
google (source organisation)
Answer access
Columns outside the prompt allowlist, including any answer or reference solution, are separated into restricted storage and never rendered.

What a problem looks like

A yes/no question and the passage it is asked against. The boolean answer is not part of the prompt.

Read an actual problem

How it is scored

The source ships the boolean answer with every question; this catalogue keeps it in restricted storage and never renders it.

Metric: accuracy over the 3,270 validation questions

Why it is hard

The questions were written by people who wanted to know the answer, not to make a test, so they rarely line up with a single sentence of the passage.

Limitations and criticism

Binary accuracy measures the chosen label without checking whether a model identified the supporting evidence or used valid reasoning. Since the relevant passage is supplied, BoolQ does not measure retrieval or the ability to find an authoritative source. Performance on a fixed public validation set also cannot by itself establish generalization to unseen search questions or rule out prior exposure to the examples.

Cite

@article{clark2019boolq,
  title={BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions},
  author={Christopher Clark and Kenton Lee and Ming-Wei Chang and Tom Kwiatkowski and Michael Collins and Kristina Toutanova},
  journal={arXiv preprint arXiv:1905.10044},
  year={2019}
}

Discussion

1 thread and 0 posts on this page. Anyone may reply; no account is required.

Contribute

Share a reproduction, report an evaluation issue, or suggest a correction to this benchmark overview.

Tasks

Whole public benchmark.

Public problems
3,270
Formats
text (3,270)
Problems with images
0
Answer availability
Answer published by the source (3,270)
Configs
default (3,270)
Splits
validation (3,270)
Awaiting a first discussion
3,270
Problems with a discussion
0
Public contributions
0

Provenance and terms

Pinned source revisions, newest first:

  • 35b264d03638db9f4ce671b711558bf7ff0f80d5 · acquired 2026-09-12 · release 9192c36c5220
  • 35b264d03638db9f4ce671b711558bf7ff0f80d5 · acquired 2026-09-12 · release fc6c571be9dd
  • 35b264d03638db9f4ce671b711558bf7ff0f80d5 · acquired 2026-09-12 · release 56068dc8733a
Licence
Not recorded. This catalogue has not verified redistribution terms.
Redistribution terms
Not recorded. This catalogue has not verified redistribution terms.
Publisher requests
Not recorded. This catalogue has not verified redistribution terms.
Terms last checked
Never