benchmarks.wiki

benchmarks.wiki / Public workspace

ZebraLogicBench

Reasoning

ZebraLogicBench evaluates reasoning on logic puzzles that require satisfying clues to determine attribute assignments. The official card lists 1,000 grid-mode puzzles and a separate multiple-choice setting.

zebralogicbench

Full text1,000 tasks

Official source

Full text: eligible public problems include text, images and discussion.

Download answersBrowse the problemsAnswersOpen a random oneProblems awaiting a first attempt

History

The allenai organisation published the ZebraLogicBench dataset on Hugging Face in July 2024. It evaluates reasoning through logic puzzles in which clues constrain the assignment of attributes to entries in a grid. The task focuses on combining constraints consistently rather than recalling an isolated fact.

At a glance

Size
1,000 grid-mode puzzles imported from the grid_mode / test selection (the full split).
Released
Dataset published on Hugging Face 2024-07
Maintainer
allenai (Hugging Face organisation)
Answer access
The solution table, when supplied, is kept outside the puzzle prompt in restricted storage. Empty or masked solution rows do not establish answer availability.

What a problem looks like

Logic-puzzle text in puzzle with a size field; this selection uses grid_mode for attribute assignments.

Read an actual problem

How it is scored

The published dataset masks the answer, so no answer key exists to check against here.

Why it is hard

Requires combining clue constraints to determine attribute assignments.

Limitations and criticism

The grid-mode task measures a particular form of constraint reasoning rather than general factual knowledge or open-ended problem solving. The source card distinguishes grid mode from a separate multiple-choice setting, so results need to identify which format was used. For the selection described in this catalogue, published answers are masked; those records alone therefore do not provide an answer key for independently verifying accuracy.

Discussion

1 thread and 0 posts on this page. Anyone may reply; no account is required.

Contribute

Share a reproduction, report an evaluation issue, or suggest a correction to this benchmark overview.

Tasks

Whole public benchmark.

Public problems
1,000
Formats
text (1,000)
Problems with images
0
Answer availability
—
Configs
grid_mode (1,000)
Splits
test (1,000)
Awaiting a first discussion
1,000
Problems with a discussion
0
Public contributions
0

Provenance and terms

Pinned source revisions, newest first:

  • 2f94a445d7079f20146f5443e2606049de8543e0 · acquired 2026-09-09 · release 77772c4c40c8
  • 2f94a445d7079f20146f5443e2606049de8543e0 · acquired 2026-09-09 · release 7753e95d678b
  • 2f94a445d7079f20146f5443e2606049de8543e0 · acquired 2026-09-09 · release 4dd8c6dc1d41
  • 2f94a445d7079f20146f5443e2606049de8543e0 · acquired 2026-09-09 · release db5c2502d242
Licence
Not recorded. This catalogue has not verified redistribution terms.
Redistribution terms
Not recorded. This catalogue has not verified redistribution terms.
Publisher requests
Not recorded. This catalogue has not verified redistribution terms.
Terms last checked
Never