benchmarks.wiki

benchmarks.wiki / Public workspace

Terminal-Bench 2.1

Agents / Code

Terminal-Bench 2.1 evaluates agents performing tasks in terminal environments. Each task supplies instructions and environment configuration, and version 2.1 is tracked separately from 2.0.

terminal-bench-2-1

Full text89 tasks

Official source

Full text: eligible public problems include text, images and discussion.

Download answersBrowse the problemsAnswersOpen a random oneProblems awaiting a first attempt

History

The harbor-framework organisation hosts Terminal-Bench 2.1 in a repository created in May 2026. This version evaluates agents carrying out tasks in terminal environments, where success requires interaction with a configured environment. It is tracked separately from Terminal-Bench 2.0, preserving the distinction between benchmark versions.

How it was built

Each task supplies an instruction file and environment configuration, with verification performed through the external task environment and grader. The catalogue selection expects 89 task records for version 2.1; this is an acquisition expectation rather than a confirmed count of imported tasks. Importing the instructions alone does not reproduce the executable evaluation.

At a glance

Size
Catalog selection for version 2.1: tasks split, expected 89 task records, source-row limit 200.
License
Apache-2.0 (declared in the GitHub repository)
Released
Repository created on GitHub 2026-05
Maintainer
harbor-framework (GitHub organisation)
Answer access
No inline answer is acquired from instruction.md; verification requires the external task environment and grader.

What a problem looks like

Each instruction.md becomes a task with an instruction field. The task also has environment configuration; the instruction import does not turn environment files into question text.

Read an actual problem

How it is scored

Verification requires the external task environment and grader.

Why it is hard

Requires performing tasks in terminal environments.

Limitations and criticism

A reported result depends on the agent, its tools and resource limits, and the configured environment as well as the underlying language model. The grader establishes success under the task checks, which need not capture every property of a robust real-world solution. Comparisons require the benchmark version and execution setup to be identified; a score for version 2.0 should not be substituted for a version 2.1 result.

Cite

@misc{merrill2026terminalbenchbenchmarkingagentshard,
      title={Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces},
      author={Mike A. Merrill and Alexander G. Shaw and Nicholas Carlini and Boxuan Li and Harsh Raj and Ivan Bercovich and Lin Shi and Jeong Yeon Shin and Thomas Walshe and E. Kelly Buchanan and Junhong Shen and Guanghao Ye and Haowei Lin and Jason Poulos and Maoyu Wang and Marianna Nezhurina and Jenia Jitsev and Di Lu and Orfeas Menis Mastromichalakis and Zhiwei Xu and Zizhao Chen and Yue Liu and Robert Zhang and Leon Liangyu Chen and Anurag Kashyap and Jan-Lucas Uslu and Jeffrey Li and Jianbo Wu and Minghao Yan and Song Bian and Vedang Sharma and Ke Sun and Steven Dillmann and Akshay Anand and Andrew Lanpouthakoun and Bardia Koopah and Changran Hu and Etash Guha and Gabriel H. S. Dreiman and Jiacheng Zhu and Karl Krauth and Li Zhong and Niklas Muennighoff and Robert Amanfu and Shangyin Tan and Shreyas Pimpalgaonkar and Tushar Aggarwal and Xiangning Lin and Xin Lan and Xuandong Zhao and Yiqing Liang and Yuanli Wang and Zilong Wang and Changzhi Zhou and David Heineman and Hange Liu and Harsh Trivedi and John Yang and Junhong Lin and Manish Shetty and Michael Yang and Nabil Omi and Negin Raoof and Shanda Li and Terry Yue Zhuo and Wuwei Lin and Yiwei Dai and Yuxin Wang and Wenhao Chai and Shang Zhou and Dariush Wahdany and Ziyu She and Jiaming Hu and Zhikang Dong and Yuxuan Zhu and Sasha Cui and Ahson Saiyed and Arinbjörn Kolbeinsson and Jesse Hu and Christopher Michael Rytting and Ryan Marten and Yixin Wang and Alex Dimakis and Andy Konwinski and Ludwig Schmidt},
      year={2026},
      eprint={2601.11868},
      archivePrefix={arXiv},
      primaryClass={cs.SE},
      url={https://arxiv.org/abs/2601.11868},
}

Discussion

1 thread and 0 posts on this page. Anyone may reply; no account is required.

Contribute

Share a reproduction, report an evaluation issue, or suggest a correction to this benchmark overview.

Tasks

Whole public benchmark.

Public problems
89
Formats
text (89)
Problems with images
0
Answer availability
Scored by the benchmark’s own harness (89)
Configs
Default (89)
Splits
tasks (89)
Awaiting a first discussion
89
Problems with a discussion
0
Public contributions
0

Provenance and terms

Pinned source revisions, newest first:

  • 7131e4375048a0e408a8fb404b5f499d726b695b · acquired 2026-09-09 · release 9bc97733b3bd
  • 7131e4375048a0e408a8fb404b5f499d726b695b · acquired 2026-09-09 · release f19a7a38c725
  • 7131e4375048a0e408a8fb404b5f499d726b695b · acquired 2026-09-09 · release e3284554d77a
  • 7131e4375048a0e408a8fb404b5f499d726b695b · acquired 2026-09-09 · release 1a25552e2327
Licence
Not recorded. This catalogue has not verified redistribution terms.
Redistribution terms
Not recorded. This catalogue has not verified redistribution terms.
Publisher requests
Not recorded. This catalogue has not verified redistribution terms.
Terms last checked
Never