benchmarks.wiki

benchmarks.wiki / Public workspace

BIG-Bench Extra Hard (BBEH)

Reasoning

BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.

bbeh

Full text4,513 tasks

Official source

Full text: eligible public problems include text, images and discussion.

Download answersBrowse the problemsAnswersOpen a random oneProblems awaiting a first attempt

History

Google DeepMind developed BIG-Bench Extra Hard as a more demanding evaluation of language-model reasoning following BIG-Bench Hard. The source repository was created in February 2025. The benchmark uses challenging task categories to probe reasoning beyond the difficulty of its predecessor.

How it was built

The repository distributes full and mini versions with examples represented as text inputs and target answers, alongside official evaluation code. This catalogue selects 200 Boolean-expression questions from the full distribution. Those records represent one category and should not be interpreted as the size or composition of the complete benchmark.

At a glance

Size
Catalog selection: the full distribution, 4,513 unique questions across all 23 task directories; 7 source rows repeat an earlier prompt and are recorded as variants. The mini distribution repeats a subset and is not imported separately.
License
Apache-2.0 (declared in the GitHub repository)
Released
Repository created on GitHub 2025-02
Maintainer
google-deepmind (GitHub organisation)
Answer access
The target field, when supplied, is separated from input into restricted answer storage.

What a problem looks like

Text prompts in the input field of examples; the selection is the full distribution across all 23 task categories.

Read an actual problem

How it is scored

The source ships an answer with every problem; this catalogue keeps it in restricted storage and never renders it. The repository ships the official evaluation code as bbeh/evaluate.py.

Why it is hard

Requires reasoning over challenging tasks; this selection focuses on Boolean expressions.

Limitations and criticism

Performance on the Boolean-expression selection measures a narrower capability than performance across the complete suite. An aggregate reported for BIG-Bench Extra Hard therefore cannot be treated as an accuracy figure for these 200 records. Comparisons also need to distinguish the full and mini distributions and use consistent answer-evaluation rules.

Cite

@article{bbeh,
      title={BIG-Bench Extra Hard},
      author={Mehran Kazemi, Bahare Fatemi, Hritik Bansal, John Palowitch, Chrysovalantis Anastasiou, Sanket Vaibhav Mehta, Lalit K. Jain, Virginia Aglietti, Disha Jindal, Peter Chen, Nishanth Dikkala, Gladys Tyen, Xin Liu, Uri Shalit, Silvia Chiappa, Kate Olszewska, Yi Tay, Vinh Q. Tran, Quoc V. Le, Orhan Firat},
      journal={arXiv preprint arXiv:2502.19187},
      year={2025},
}

Discussion

1 thread and 0 posts on this page. Anyone may reply; no account is required.

Contribute

Share a reproduction, report an evaluation issue, or suggest a correction to this benchmark overview.

Tasks

Whole public benchmark.

Public problems
4,513
Formats
text (4,513)
Problems with images
0
Answer availability
Answer published by the source (4,511)
Configs
bbeh_boardgame_qa (200) · bbeh_boolean_expressions (200) · bbeh_buggy_tables (200) · bbeh_causal_understanding (200) · bbeh_disambiguation_qa (120) · bbeh_dyck_languages (200) · bbeh_geometric_shapes (200) · bbeh_hyperbaton (200) · bbeh_linguini (194) · bbeh_movie_recommendation (200) · bbeh_multistep_arithmetic (200) · bbeh_nycc (200) · bbeh_object_counting (200) · bbeh_object_properties (200) · bbeh_sarc_triples (200) · bbeh_shuffled_objects (200) · bbeh_spatial_reasoning (199) · bbeh_sportqa (200) · bbeh_temporal_sequence (200) · bbeh_time_arithmetic (200) · bbeh_web_of_lies (200) · bbeh_word_sorting (200) · bbeh_zebra_puzzles (200)
Splits
full (4,513)
Awaiting a first discussion
4,513
Problems with a discussion
0
Public contributions
0

Provenance and terms

Pinned source revisions, newest first:

  • 80d12ca916b7158f22293fcf3144f4d3d854d4be · acquired 2026-09-23 · release 7da5c6a85d8d
  • 80d12ca916b7158f22293fcf3144f4d3d854d4be · acquired 2026-09-09 · release dfbc3f2ad31e
  • 80d12ca916b7158f22293fcf3144f4d3d854d4be · acquired 2026-09-09 · release 5971a9aa3df4
  • 80d12ca916b7158f22293fcf3144f4d3d854d4be · acquired 2026-09-09 · release 22a6c6a76f02
  • 80d12ca916b7158f22293fcf3144f4d3d854d4be · acquired 2026-09-08 · release 27dd7d5361f8
Licence
Not recorded. This catalogue has not verified redistribution terms.
Redistribution terms
Not recorded. This catalogue has not verified redistribution terms.
Publisher requests
Not recorded. This catalogue has not verified redistribution terms.
Terms last checked
Never