benchmarks.wiki

benchmarks.wiki / Public workspace

HellaSwag

Reasoning

HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.

hellaswag

Full text10,042 tasks

Official source

Full text: eligible public problems include text, images and discussion.

Download answersBrowse the problemsAnswersOpen a random oneProblems awaiting a first attempt

History

Rowan Zellers and collaborators at the University of Washington and the Allen Institute for AI introduced HellaSwag in 2019 to test commonsense inference about everyday situations. They sought examples that people could resolve easily but that language models could not solve reliably through surface plausibility alone. The March 2022 Hugging Face publication date refers to the hosted dataset distribution, not the original benchmark release.

How it was built

HellaSwag draws contexts and human-written continuations from ActivityNet video captions and WikiHow descriptions of activities. Language models generate candidate incorrect endings, and adversarial filtering selects distractors that are difficult for discriminative models to reject. Human validation checks the resulting multiple-choice examples. Each item offers four continuations; the validation split contains 10,042 items, of which this catalogue imports 200.

At a glance

Metric
accuracy over the 10,042 validation items
Size
10,042 problems imported from the default / validation selection (the full validation split).
License
mit (declared in the dataset card’s Licensing Information section, which points at the project’s MIT LICENSE)
Released
Dataset published on Hugging Face 2022-03
Maintainer
Rowan (source organisation)
Answer access
Columns outside the prompt allowlist, including any answer or reference solution, are separated into restricted storage and never rendered.
Highest recorded
88.9 Acc. — DeepSeek-V3 (vendor, self-reported)

What a problem looks like

An activity label and a context sentence, with the four candidate continuations as a list. Which one is correct is not part of the prompt.

Read an actual problem

How it is scored

The source ships the index of the correct continuation; this catalogue keeps it in restricted storage and never renders it.

Metric: accuracy over the 10,042 validation items

Reported scores

Evaluation conditions differ between rows, so this is not a ranking. Rows marked vendor are reports by the model’s own organization.

ModelScoreMetricSetupReported bySource
DeepSeek-V3DeepSeek88.9Acc.Base model; 10-shot.vendor (self-reported)Base Model, Standard Benchmarks, HellaSwag (Acc.), final column

Why it is hard

The distractors were selected specifically because models rank them highly, so the gap it measures is between surface plausibility and actual commonsense.

Limitations and criticism

The task tests selection among supplied endings rather than the ability to generate an appropriate continuation or act in the described situation. Its adversarial filtering reflects the models used during construction and does not guarantee equal difficulty for later systems. The public dataset presents a potential contamination risk, while differences in prompting and candidate-scoring procedures can complicate comparisons between reports.

How much room is left

The vendor result recorded here reports 88.9% accuracy for the DeepSeek-V3 base model with 10-shot prompting. That is high performance but remains below perfect accuracy on the reported evaluation. This single result does not establish that HellaSwag is saturated; comparisons near the upper end need to keep the split and evaluation procedure consistent.

Cite

@article{zellers2019hellaswag,
  title={HellaSwag: Can a Machine Really Finish Your Sentence?},
  author={Rowan Zellers and Ari Holtzman and Yonatan Bisk and Ali Farhadi and Yejin Choi},
  journal={arXiv preprint arXiv:1905.07830},
  year={2019}
}

Discussion

1 thread and 0 posts on this page. Anyone may reply; no account is required.

Contribute

Share a reproduction, report an evaluation issue, or suggest a correction to this benchmark overview.

Tasks

Whole public benchmark.

Public problems
10,042
Formats
text (10,042)
Problems with images
0
Answer availability
Answer published by the source (10,042)
Configs
default (10,042)
Splits
validation (10,042)
Awaiting a first discussion
10,042
Problems with a discussion
0
Public contributions
0

Provenance and terms

Pinned source revisions, newest first:

  • 218ec52e09a7e7462a5400043bb9a69a41d06b76 · acquired 2026-09-12 · release 7e083cabda9a
  • 218ec52e09a7e7462a5400043bb9a69a41d06b76 · acquired 2026-09-12 · release da7444a35989
  • 218ec52e09a7e7462a5400043bb9a69a41d06b76 · acquired 2026-09-12 · release 7fb6311c6df2
Licence
Not recorded. This catalogue has not verified redistribution terms.
Redistribution terms
Not recorded. This catalogue has not verified redistribution terms.
Publisher requests
Not recorded. This catalogue has not verified redistribution terms.
Terms last checked
Never