benchmarks.wiki

benchmarks.wiki / Public workspace

RULER

Reasoning / Long context

RULER is a synthetic benchmark for the context length a model can actually use. It generates retrieval, multi-hop tracing, aggregation and question-answering tasks at any length, exposing the gap against the advertised window.

ruler

Official source

Download answersVisit the official benchmarkWhere the answers areStart a benchmark discussion

History

NVIDIA researchers introduced RULER in 2024 to investigate how much of an advertised context window a language model could actually use. The benchmark broadened long-context testing beyond simple retrieval to include tracing, aggregation and question answering. Its purpose was to expose task-dependent failures that a nominal maximum input length does not describe.

How it was built

RULER generates synthetic long documents containing information that must be retrieved, traced or aggregated to answer a task. Generation provides the corresponding answer key and allows evaluations at different context lengths instead of relying on one fixed document collection. The benchmark reports accuracy by task and length, with effective context length defined through a performance threshold.

At a glance

Metric
accuracy per task at each context length, and the effective length at which performance falls below a threshold
Size
Generated per run; nothing fixed to import.
License
apache-2.0 (declared by the LICENSE file in the official NVIDIA/RULER repository)
Maintainer
NVIDIA (source organisation)
Answer access
No problem text is imported, so there is no answer storage for this entry. The source distributes the answers under its own terms.

What a problem looks like

Synthetic long documents with planted information to retrieve, trace, or aggregate.

How it is scored

Generated with its own answer key; nothing is imported here.

Metric: accuracy per task at each context length, and the effective length at which performance falls below a threshold

Why it is hard

It separates the advertised context window from the usable one. Most models degrade well before the limit they claim, and the point of failure depends on the task, not just the length.

Limitations and criticism

Synthetic contexts isolate particular information-processing demands but do not reproduce the full complexity of long documents or sustained real-world workflows. An effective-length result depends on the selected tasks, generation settings and threshold, rather than being an intrinsic constant of a model. Strong performance on a simple retrieval task therefore does not establish equivalent ability to combine or trace information over the same context length.

Cite

@article{hsieh2024ruler,
  title={RULER: What's the Real Context Size of Your Long-Context Language Models?},
  author={Cheng-Ping Hsieh and Simeng Sun and Samuel Kriman and Shantanu Acharya and Dima Rekesh and Fei Jia and Yang Zhang and Boris Ginsburg},
  journal={arXiv preprint arXiv:2404.06654},
  year={2024}
}

Discussion

1 thread and 0 posts on this page. Anyone may reply; no account is required.

Contribute

Share a reproduction, report an evaluation issue, or suggest a correction to this benchmark overview.

Public problem summary

Catalogue metadata only; public problem statistics are not available.

Public problems
—
Formats
—
Problems with images
—
Answer availability
—
Configs
—
Splits
—
Awaiting a first discussion
—
Problems with a discussion
—
Public contributions
—