benchmarks.wiki

benchmarks.wiki / Public workspace

BIG-Bench Hard

Reasoning

BIG-Bench Hard is the set of 23 BIG-Bench tasks on which models of the time scored below the average human rater. It is best known for showing how much chain-of-thought prompting changes the result.

bbh

Full text6,507 tasks

Official source

Full text: eligible public problems include text, images and discussion.

Download answersBrowse the problemsAnswersOpen a random oneProblems awaiting a first attempt

History

BIG-Bench Hard was introduced in 2022 by Mirac Suzgun and collaborators to examine reasoning tasks that remained difficult for language models despite increased scale. The group selected 23 BIG-Bench tasks on which the evaluated models fell below average human performance. Their study examined how chain-of-thought prompting changed performance on this difficult subset.

How it was built

BBH reuses a selected subset of BIG-Bench rather than collecting a uniform new set of questions. Its tasks include boolean expressions, date understanding, object counting, and logical deduction, with task-specific targets and answer formats. The source described in the catalogue contains about 6,500 examples across the tasks.

At a glance

Size
6,507 unique problems imported from the 27 task files of the official repository (6,511 rows; 4 rows repeat an earlier prompt and are recorded as variants).
License
mit (declared on the Hugging Face dataset card)
Released
Dataset published on Hugging Face 2023-09
Maintainer
maveriq (source organisation)
Answer access
The target field of every example is separated from the input into restricted answer storage and never rendered.
Highest recorded
87.5 EM — DeepSeek-V3 (vendor, self-reported)

What a problem looks like

Varies by task: boolean expressions, date understanding, object counting, logical deduction and others.

Read an actual problem

How it is scored

Each task ships its own target with every problem; this catalogue keeps it in restricted storage and never renders it. The repository ships the official evaluation code.

Reported scores

Evaluation conditions differ between rows, so this is not a ranking. Rows marked vendor are reports by the model’s own organization.

ModelScoreMetricSetupReported bySource
DeepSeek-V3DeepSeek87.5EMBase model; 3-shot.vendor (self-reported)Base Model, Standard Benchmarks, BBH (EM), final column

Why it is hard

These are exactly the tasks that resisted scale, and several of them need an explicit multi-step procedure rather than a single judgement.

Limitations and criticism

The hard designation reflects the models and evaluation conditions used for the original selection, rather than a permanent difficulty threshold. Task formats differ, so an aggregate score can conceal uneven abilities across the 23 tasks. Prompt examples, chain-of-thought instructions, and final-answer extraction affect comparability, while the public questions and targets create a potential route for training-data overlap.

How much room is left

The catalogue records a vendor-reported exact-match score of 87.5 for the DeepSeek-V3 base model with three-shot prompting. This result leaves errors under that setup, but it is neither a complete leaderboard nor evidence of the current performance ceiling. Per-task results and matched prompting conditions remain necessary to determine whether a particular part of BBH still separates systems.

Cite

@article{suzgun2022bbh,
  title={Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them},
  author={Mirac Suzgun and Nathan Scales and Nathanael Schärli and Sebastian Gehrmann and Yi Tay and Hyung Won Chung and Aakanksha Chowdhery and Quoc V. Le and Ed H. Chi and Denny Zhou and Jason Wei},
  journal={arXiv preprint arXiv:2210.09261},
  year={2022}
}

Discussion

1 thread and 0 posts on this page. Anyone may reply; no account is required.

Contribute

Share a reproduction, report an evaluation issue, or suggest a correction to this benchmark overview.

Tasks

Whole public benchmark.

Public problems
6,507
Formats
text (6,507)
Problems with images
0
Answer availability
Answer published by the source (6,507)
Configs
boolean_expressions (250) · causal_judgement (185) · date_understanding (250) · disambiguation_qa (250) · dyck_languages (250) · formal_fallacies (250) · geometric_shapes (250) · hyperbaton (250) · logical_deduction_five_objects (250) · logical_deduction_seven_objects (250) · logical_deduction_three_objects (250) · movie_recommendation (250) · multistep_arithmetic_two (250) · navigate (250) · object_counting (250) · penguins_in_a_table (146) · reasoning_about_colored_objects (250) · ruin_names (250) · salient_translation_error_detection (250) · snarks (178) · sports_understanding (248) · temporal_sequences (250) · tracking_shuffled_objects_five_objects (250) · tracking_shuffled_objects_seven_objects (250) · tracking_shuffled_objects_three_objects (250) · web_of_lies (250) · word_sorting (250)
Splits
main (6,507)
Awaiting a first discussion
6,507
Problems with a discussion
0
Public contributions
0

Provenance and terms

Pinned source revisions, newest first:

  • 9ee07bd481feebf959a6b59d61ea57bdcf30964d · acquired 2026-09-23 · release af4232583def
  • catalog-20-381f44295965a2e9
Licence
Not recorded. This catalogue has not verified redistribution terms.
Redistribution terms
Not recorded. This catalogue has not verified redistribution terms.
Publisher requests
Not recorded. This catalogue has not verified redistribution terms.
Terms last checked
Never