benchmarks.wiki

benchmarks.wiki / Public workspace

WinoGrande

Reasoning

WinoGrande scales the Winograd Schema Challenge to 44,000 problems in which a pronoun must be resolved between two sentences that differ by one word. It was built through a procedure that strips dataset-side bias.

winogrande

Official source

Download answersVisit the official benchmarkWhere the answers areStart a benchmark discussion

History

The Allen Institute for Artificial Intelligence introduced WinoGrande in 2019 to scale up commonsense language understanding evaluation inspired by the Winograd Schema Challenge. The project sought a large collection with fewer exploitable statistical cues than ordinary crowdsourced data. The March 2022 Hugging Face publication recorded here refers to a later distribution.

How it was built

WinoGrande was built from crowdsourced sentence-completion problems and subjected to adversarial filtering to reduce dataset biases. The source contains about 44,000 problems, each presenting a sentence with a blank and two candidate fillers. Evaluation protocols include different amounts of training data, ranging from 160 to 40,000 examples.

At a glance

Metric
accuracy, reported across training-set sizes from 160 to 40,000 examples
Size
About 44,000 problems in the source; none imported.
License
The dataset card has a Licensing Information section but names no licence.
Released
Dataset published on Hugging Face 2022-03
Maintainer
allenai (source organisation)
Answer access
No problem text is imported, so there is no answer storage for this entry. The source distributes the answers under its own terms.
Highest recorded
84.9 Acc. — DeepSeek-V3 (vendor, self-reported)

What a problem looks like

A sentence with a blank and two candidate fillers.

How it is scored

The source ships the correct option; nothing is imported here.

Metric: accuracy, reported across training-set sizes from 160 to 40,000 examples

Reported scores

Evaluation conditions differ between rows, so this is not a ranking. Rows marked vendor are reports by the model’s own organization.

ModelScoreMetricSetupReported bySource
DeepSeek-V3DeepSeek84.9Acc.Base model; 5-shot.vendor (self-reported)Base Model, Standard Benchmarks, WinoGrande (Acc.), final column

Why it is hard

The two sentences differ by a single word, so every cue except the intended commonsense one is held constant.

Limitations and criticism

Adversarial filtering reduces cues detectable by the filtering procedure; it does not establish that all statistical shortcuts have been removed. Binary sentence completion measures a constrained form of commonsense language understanding rather than open-ended reasoning about the world. Scores also depend on the amount of task-specific training data, so those protocols should be distinguished from few-shot prompting.

How much room is left

The recorded vendor report gives DeepSeek-V3 base 84.9 percent accuracy with five examples in the prompt. That result leaves headroom under the stated protocol, although the two-choice format has a 50 percent uniform-guessing baseline. A single few-shot report does not establish the best achievable result under the supervised training protocols.

Cite

@article{sakaguchi2019winogrande,
  title={WinoGrande: An Adversarial Winograd Schema Challenge at Scale},
  author={Keisuke Sakaguchi and Ronan Le Bras and Chandra Bhagavatula and Yejin Choi},
  journal={arXiv preprint arXiv:1907.10641},
  year={2019}
}

Discussion

1 thread and 0 posts on this page. Anyone may reply; no account is required.

Contribute

Share a reproduction, report an evaluation issue, or suggest a correction to this benchmark overview.

Public problem summary

Catalogue metadata only; public problem statistics are not available.

Public problems
—
Formats
—
Problems with images
—
Answer availability
—
Configs
—
Splits
—
Awaiting a first discussion
—
Problems with a discussion
—
Public contributions
—