benchmarks.wiki

benchmarks.wiki / Public workspace

TruthfulQA

Reasoning / Safety

TruthfulQA measures whether a model repeats falsehoods that people commonly believe. Its 817 questions span 38 categories including health, law, finance and politics, and each one is written to invite a plausible wrong answer.

truthfulqa

Full text817 tasks

Official source

Full text: eligible public problems include text, images and discussion.

Download answersBrowse the problemsAnswersOpen a random oneProblems awaiting a first attempt

History

Stephanie Lin, Jacob Hilton, and Owain Evans introduced TruthfulQA in 2021 to investigate whether language models reproduce common human falsehoods. The benchmark targeted the possibility that learning to imitate human text also teaches a model to repeat misconceptions. Its research question concerned truthfulness rather than merely producing plausible or fluent answers.

How it was built

The authors wrote questions intended to invite plausible false answers associated with common misconceptions. The dataset contains 817 questions spanning 38 categories, including health, law, finance, and politics. Reference material includes a best answer and lists of correct and incorrect answers. This catalogue selects 200 records from the generation configuration and keeps those references outside the prompt.

At a glance

Metric
truthfulness and informativeness, judged per answer; the multiple-choice configuration reports MC1/MC2 accuracy
Size
817 problems imported from the generation / validation selection (the full validation split). The multiple-choice configuration covers the same questions and is not imported separately.
License
apache-2.0 (declared on the Hugging Face dataset card)
Released
Dataset published on Hugging Face 2022-06
Maintainer
truthfulqa (source organisation)
Answer access
Columns outside the prompt allowlist, including any answer or reference solution, are separated into restricted storage and never rendered.

What a problem looks like

One question, written to invite a plausible falsehood. Its category, its adversarial type and the source documenting the misconception all narrow the answer, so none of them is published.

Read an actual problem

How it is scored

The source ships a best answer plus lists of correct and incorrect answers; this catalogue keeps all of them in restricted storage and never renders them.

Metric: truthfulness and informativeness, judged per answer; the multiple-choice configuration reports MC1/MC2 accuracy

Why it is hard

Every question has an answer that sounds right because people repeat it. Scale alone tends to make this worse, because the falsehood is well represented in the training text.

Limitations and criticism

TruthfulQA is deliberately focused on misconception-eliciting questions, so its score does not estimate the frequency of false statements in ordinary use. Generation evaluations distinguish truthfulness from informativeness because an uninformative answer can avoid making a false claim. The multiple-choice configurations and free-form generation assess different behavior and should not be treated as interchangeable metrics. Public questions and reference answers also make training exposure a concern when interpreting later results.

Cite

@article{lin2021truthfulqa,
  title={TruthfulQA: Measuring How Models Mimic Human Falsehoods},
  author={Stephanie Lin and Jacob Hilton and Owain Evans},
  journal={arXiv preprint arXiv:2109.07958},
  year={2021}
}

Discussion

1 thread and 0 posts on this page. Anyone may reply; no account is required.

Contribute

Share a reproduction, report an evaluation issue, or suggest a correction to this benchmark overview.

Tasks

Whole public benchmark.

Public problems
817
Formats
text (817)
Problems with images
0
Answer availability
Answer published by the source (817)
Configs
generation (817)
Splits
validation (817)
Awaiting a first discussion
817
Problems with a discussion
0
Public contributions
0

Provenance and terms

Pinned source revisions, newest first:

  • 741b8276f2d1982aa3d5b832d3ee81ed3b896490 · acquired 2026-09-12 · release 7e98884f0664
  • 741b8276f2d1982aa3d5b832d3ee81ed3b896490 · acquired 2026-09-12 · release 28ecf922883b
Licence
Not recorded. This catalogue has not verified redistribution terms.
Redistribution terms
Not recorded. This catalogue has not verified redistribution terms.
Publisher requests
Not recorded. This catalogue has not verified redistribution terms.
Terms last checked
Never