benchmarks.wiki

benchmarks.wiki / Public workspace

AssistantBench

Agents / Search

AssistantBench collects realistic, time-consuming tasks that can only be answered by working through the live web, written so that each has one determinable answer. It measures whether a web agent is actually useful.

assistantbench

Full text181 tasks

Official source

Full text: eligible public problems include text, images and discussion.

Download answersBrowse the problemsAnswersOpen a random oneProblems awaiting a first attempt

History

The AssistantBench research team released its dataset in July 2024 to evaluate assistants on realistic, time-consuming web tasks. Its motivating question was whether an agent could assemble a determinate answer through substantial search across multiple sources, rather than merely retrieve a fact from one page.

How it was built

The benchmark collects tasks designed to have a determinable answer even when obtaining it requires work across websites. The source test selection contains 181 tasks and withholds their answers; the catalog imports all 181 task statements. Official evaluation uses a submission service and reports accuracy and answer rate on the held-out test set.

At a glance

Metric
accuracy and answer rate over the held-out test set, scored by the official submission service
Size
All 181 tasks imported from the default / test selection, with a source-row limit of 200.
License
apache-2.0 (declared on the Hugging Face dataset card)
Released
Dataset published on Hugging Face 2024-07
Maintainer
AssistantBench (source organisation)
Answer access
Columns outside the prompt allowlist, including any answer or reference solution, are separated into restricted storage and never rendered.

What a problem looks like

One task statement. The evaluation subset, the answer, the gold source URL and the explanation are not part of the prompt.

Read an actual problem

How it is scored

The test split ships no answer at all — the source withholds it — so this catalogue has nothing to store or render for it.

Metric: accuracy and answer rate over the held-out test set, scored by the official submission service

Why it is hard

The answer is not on any single page. It has to be assembled across sites, and the search itself is most of the work.

Limitations and criticism

Performance depends on the agent's search and browsing tools as well as the underlying model. Because tasks use the live web, changes in page availability and content can change the evidence accessible during an evaluation. Accuracy and answer rate describe different aspects of performance: a result needs to establish both how often an agent answers and how often it is correct.

Cite

@article{yoran2024assistantbench,
  title={AssistantBench: Can Web Agents Solve Realistic and Time-Consuming Tasks?},
  author={Ori Yoran and Samuel Joseph Amouyal and Chaitanya Malaviya and Ben Bogin and Ofir Press and Jonathan Berant},
  journal={arXiv preprint arXiv:2407.15711},
  year={2024}
}

Discussion

1 thread and 0 posts on this page. Anyone may reply; no account is required.

Contribute

Share a reproduction, report an evaluation issue, or suggest a correction to this benchmark overview.

Tasks

Whole public benchmark.

Public problems
181
Formats
text (181)
Problems with images
0
Answer availability
—
Configs
default (181)
Splits
test (181)
Awaiting a first discussion
181
Problems with a discussion
0
Public contributions
0

Provenance and terms

Pinned source revisions, newest first:

  • 482cbbc0400f6d048438c4021727f21a10cbff49 · acquired 2026-09-12 · release 02c88c74db5a
  • 482cbbc0400f6d048438c4021727f21a10cbff49 · acquired 2026-09-12 · release 868936e7972d
  • 482cbbc0400f6d048438c4021727f21a10cbff49 · acquired 2026-09-12 · release 75ac7aefce1d
Licence
Not recorded. This catalogue has not verified redistribution terms.
Redistribution terms
Not recorded. This catalogue has not verified redistribution terms.
Publisher requests
Not recorded. This catalogue has not verified redistribution terms.
Terms last checked
Never