benchmarks.wiki

benchmarks.wiki / Public workspace

LongBench v2

Reasoning / Long context

LongBench v2 evaluates deep understanding and reasoning over long contexts through multiple-choice questions. Its official description lists 503 questions spanning tasks such as single-document and multi-document QA and code-repository understanding.

longbench-v2

Full text200 tasks

Official source

Full text: eligible public problems include text, images and discussion.

Download answersBrowse the problemsAnswersOpen a random oneProblems awaiting a first attempt

History

LongBench v2 was introduced in 2024 as a benchmark for deep understanding and reasoning over long contexts. The dataset distributed by zai-org on Hugging Face was published in December 2024. It addresses tasks such as single-document and multi-document question answering and code-repository understanding, extending evaluation beyond merely locating a fact in a long input.

At a glance

Metric
accuracy over multiple-choice questions (README: 503 multiple-choice questions)
Size
503 questions described in the catalog introduction; this train selection expects 200 task records with a source-row limit of 200.
License
apache-2.0 (declared on the Hugging Face dataset card)
Released
Dataset published on Hugging Face 2024-12
Maintainer
zai-org (Hugging Face organisation)
Answer access
Fields outside the prompt allowlist, including answers when supplied, remain in restricted storage; the converter alone does not establish whether usable answers are available.

What a problem looks like

Multiple-choice questions with context, choice_A through choice_D, and optional domain, sub_domain, difficulty, and length fields.

Read an actual problem

How it is scored

For the 200 imported problems, the source ships an answer with every problem; this catalogue keeps it in restricted storage and never renders it. Automated evaluation is provided by the LongBench repository at https://github.com/THUDM/LongBench.

Metric: accuracy over multiple-choice questions (README: 503 multiple-choice questions)

Why it is hard

Requires deep understanding and reasoning over long contexts, including single-document and multi-document QA and code-repository understanding.

Limitations and criticism

The four-option format measures answer selection rather than the ability to produce a supported explanation or a complete working change to a code repository. A correct selection does not reveal which parts of the context supported it. Results also depend on how an evaluation handles contexts that exceed a model's input limit. The official collection contains 503 questions; this catalog's 200 imported records are a limited selection and should not be confused with a full-benchmark evaluation.

Cite

@article{bai2024longbench2,
  title={LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks}, 
  author={Yushi Bai and Shangqing Tu and Jiajie Zhang and Hao Peng and Xiaozhi Wang and Xin Lv and Shulin Cao and Jiazheng Xu and Lei Hou and Yuxiao Dong and Jie Tang and Juanzi Li},
  journal={arXiv preprint arXiv:2412.15204},
  year={2024}
}

Discussion

1 thread and 0 posts on this page. Anyone may reply; no account is required.

Contribute

Share a reproduction, report an evaluation issue, or suggest a correction to this benchmark overview.

Tasks

Whole public benchmark.

Public problems
200
Formats
text (200)
Problems with images
0
Answer availability
Answer published by the source (200)
Configs
Default (200)
Splits
train (200)
Awaiting a first discussion
200
Problems with a discussion
0
Public contributions
0

Provenance and terms

Pinned source revisions, newest first:

  • 2b48e494f2c7a2f0af81aae178e05c7e1dde0fe9 · acquired 2026-09-09 · release 163e4eea78f9
  • 2b48e494f2c7a2f0af81aae178e05c7e1dde0fe9 · acquired 2026-09-09 · release 2154326d7cb8
  • 2b48e494f2c7a2f0af81aae178e05c7e1dde0fe9 · acquired 2026-09-09 · release 052dc8a10560
  • 2b48e494f2c7a2f0af81aae178e05c7e1dde0fe9 · acquired 2026-09-09 · release 0a6d2989e8ed
Licence
Not recorded. This catalogue has not verified redistribution terms.
Redistribution terms
Not recorded. This catalogue has not verified redistribution terms.
Publisher requests
Not recorded. This catalogue has not verified redistribution terms.
Terms last checked
Never