benchmarks.wiki

benchmarks.wiki / Public workspace

IFEval

Reasoning / Instruction following

IFEval measures whether a model follows verifiable instructions -- word counts, formats, forbidden words -- rather than whether an answer is good. Every constraint is checked programmatically.

ifeval

Full text541 tasks

Official source

Full text: eligible public problems include text, images and discussion.

Download answersBrowse the problemsAnswersOpen a random oneProblems awaiting a first attempt

History

Google researchers introduced IFEval in 2023 to measure instruction following with constraints that can be checked automatically. The motivation was to make compliance with explicit instructions measurable without relying entirely on subjective judgments of answer quality. The Google dataset was published on Hugging Face in December 2023.

How it was built

IFEval prompts contain one or more verifiable constraints, such as requirements about word counts, output format or the presence of particular words. Constraint identifiers and associated checking information accompany the natural-language prompts so that compliance can be assessed programmatically. This catalog imports 200 prompts from the source selection labeled train; that import count is not the size of the complete benchmark.

At a glance

Size
541 problems imported from the default / train selection (the full split; the source names its only split train).
License
apache-2.0 (declared on the source)
Released
Dataset published on Hugging Face 2023-12
Maintainer
google (source organisation)
Answer access
Columns outside the prompt allowlist, including any answer or reference solution, are separated into restricted storage and never rendered.

What a problem looks like

A natural-language prompt carrying one or more machine-checkable constraints, with the constraint identifiers recorded alongside it.

Read an actual problem

How it is scored

Every column outside the prompt is separated into restricted storage, but the converter does not interpret it, so this catalogue does not claim a usable answer key exists.

Why it is hard

The constraints are easy to state and easy to violate in passing; models routinely answer well while breaking the format they were told to use.

Limitations and criticism

Passing formal constraints does not establish that an answer is accurate, relevant or useful. Conversely, an otherwise good answer can fail because it violates a requested format. Reported results must distinguish instruction-level from prompt-level scoring, since satisfying some constraints is different from satisfying every constraint in a prompt. Strict and loose checking variants can also produce different scores and should not be treated as interchangeable.

Cite

@misc{zhou2023instructionfollowingevaluationlargelanguage,
      title={Instruction-Following Evaluation for Large Language Models}, 
      author={Jeffrey Zhou and Tianjian Lu and Swaroop Mishra and Siddhartha Brahma and Sujoy Basu and Yi Luan and Denny Zhou and Le Hou},
      year={2023},
      eprint={2311.07911},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2311.07911}, 
}

Discussion

1 thread and 0 posts on this page. Anyone may reply; no account is required.

Contribute

Share a reproduction, report an evaluation issue, or suggest a correction to this benchmark overview.

Tasks

Whole public benchmark.

Public problems
541
Formats
text (541)
Problems with images
0
Answer availability
—
Configs
default (541)
Splits
train (541)
Awaiting a first discussion
541
Problems with a discussion
0
Public contributions
0

Provenance and terms

Pinned source revisions, newest first:

  • 966cd89545d6b6acfd7638bc708b98261ca58e84 · acquired 2026-09-11 · release 54f3548aa995
  • 966cd89545d6b6acfd7638bc708b98261ca58e84 · acquired 2026-09-11 · release a7368be0a38a
  • 966cd89545d6b6acfd7638bc708b98261ca58e84 · acquired 2026-09-11 · release dd89911e924b
  • 966cd89545d6b6acfd7638bc708b98261ca58e84 · acquired 2026-09-11 · release 626f175959b6
Licence
Not recorded. This catalogue has not verified redistribution terms.
Redistribution terms
Not recorded. This catalogue has not verified redistribution terms.
Publisher requests
Not recorded. This catalogue has not verified redistribution terms.
Terms last checked
Never