benchmarks.wiki

benchmarks.wiki / Public workspace

BrowseComp-Plus

Agents / Search

BrowseComp-Plus supports evaluation of retrieval methods using BrowseComp tasks and a search corpus. This catalog links to the official distribution without republishing plaintext questions or corpus content.

browsecomp-plus

Official source

Download answersVisit the official benchmarkWhere the answers areStart a benchmark discussion

History

BrowseComp-Plus adapted BrowseComp tasks for retrieval evaluation against a search corpus. The Tevatron organisation published the dataset on Hugging Face in August 2025. Pairing tasks with a fixed corpus makes it possible to study search and answering within a specified collection of documents.

At a glance

License
mit (declared on the Hugging Face dataset card)
Released
Dataset published on Hugging Face 2025-08
Maintainer
Tevatron (Hugging Face organisation)
Answer access
This catalog does not republish plaintext questions or corpus content. Its acquisition record reports HTTP 401 for the old source; the official dataset link points to Tevatron/browsecomp-plus.
Highest recorded
55.9 Accuracy (%) — gpt-5 (paper)

What a problem looks like

BrowseComp tasks paired with a search corpus for retrieval evaluation.

Reported scores

Evaluation conditions differ between rows, so this is not a ranking. Rows marked vendor are reports by the model’s own organization.

ModelScoreMetricSetupReported bySource
gpt-555.9Accuracy (%)BM25 retriever; same tool-use prompt; fixed BrowseComp-Plus corpus.paperTable 1, BM25 rows
o349.28Accuracy (%)BM25 retriever; same tool-use prompt; fixed BrowseComp-Plus corpus.paperTable 1, BM25 rows
gpt-4.114.58Accuracy (%)BM25 retriever; same tool-use prompt; fixed BrowseComp-Plus corpus.paperTable 1, BM25 rows

Why it is hard

Requires retrieving information for BrowseComp tasks from a search corpus.

Limitations and criticism

A result on a fixed search corpus describes performance within that collection, rather than unrestricted browsing of the changing web. End-to-end answer accuracy combines retrieval, tool use and answer generation, so it does not isolate the quality of the language model or retriever alone. Comparisons require the same corpus and a clearly specified retrieval setup.

How much room is left

The paper results recorded here report 55.9% accuracy for GPT-5, 49.28% for o3 and 14.58% for GPT-4.1, using a BM25 retriever, the same tool-use prompt and the fixed BrowseComp-Plus corpus. These results leave substantial headroom in that configuration. They do not establish a ceiling across other retrieval methods or agent configurations.

Cite

@article{chen2025BrowseCompPlus,
      title={BrowseComp-Plus: A More Fair and Transparent Evaluation Benchmark of Deep-Research Agent}, 
      author={Chen, Zijian and Ma, Xueguang and Zhuang, Shengyao and Nie, Ping and Zou, Kai and Liu, Andrew and Green, Joshua and Patel, Kshama and Meng, Ruoxi and Su, Mingyi and Sharifymoghaddam, Sahel and Li, Yanxi and Hong, Haoran and Shi, Xinyu and Liu, Xuye and Thakur, Nandan and Zhang, Crystina and Gao, Luyu and Chen, Wenhu and Lin, Jimmy},
      year={2025},
      journal={arXiv preprint arXiv:2508.06600}
}

Discussion

1 thread and 0 posts on this page. Anyone may reply; no account is required.

Contribute

Share a reproduction, report an evaluation issue, or suggest a correction to this benchmark overview.

Public problem summary

Catalogue metadata only; public problem statistics are not available.

Public problems
—
Formats
—
Problems with images
—
Answer availability
—
Configs
—
Splits
—
Awaiting a first discussion
—
Problems with a discussion
—
Public contributions
—