benchmarks.wiki

benchmarks.wiki / Public workspace

WebArena

Agents / Search

WebArena gives an agent self-hosted clones of a shopping site, a forum, a wiki and a code host. It scores whether a natural-language goal was achieved on the live sites.

webarena

Official source

Download answersVisit the official benchmarkWhere the answers areStart a benchmark discussion

History

The WebArena research team introduced the benchmark in 2023 to evaluate agents carrying out natural-language goals through web interfaces. The official web-arena-x repository was created in July 2023. Self-hosted applications made it possible to evaluate multi-step navigation and interaction in a controlled web environment.

How it was built

WebArena uses self-hosted web applications representing settings such as shopping, discussion forums and code hosting, with supporting information resources. Human-authored tasks specify goals that agents must accomplish by interacting with those applications. Programmatic evaluators check task outcomes, including the resulting application state, rather than requiring the agent to follow a single reference sequence of clicks.

At a glance

Size
Not imported here; see the official distribution for the current size.
License
Apache-2.0 (declared on the source)
Released
Repository created on GitHub 2023-07
Maintainer
web-arena-x (source organisation)
Answer access
Tasks run against self-hosted web applications and are graded by the resulting state, so there is no static problem text to republish.

What a problem looks like

A natural-language goal on a self-hosted web application, with a programmatic check of the resulting state.

How it is scored

Tasks run against self-hosted web applications and are graded by the resulting state, so there is no static problem text to republish.

Why it is hard

Long horizons on real interfaces: navigation, forms and search all have to work before the goal state can be reached.

Limitations and criticism

The self-hosted environment provides controlled tasks but does not reproduce the full variability of the live web, including changing layouts, authentication barriers and service outages. A success rate also depends on the browser interface, available observations, action limits and the agent framework. Programmatic outcome checks cover specified conditions rather than every aspect of usability or safe interaction. Environment versions and reset procedures therefore matter when comparing published runs.

Cite

@article{zhou2023webarena,
  title={WebArena: A Realistic Web Environment for Building Autonomous Agents},
  author={Zhou, Shuyan and Xu, Frank F and Zhu, Hao and Zhou, Xuhui and Lo, Robert and Sridhar, Abishek and Cheng, Xianyi and Bisk, Yonatan and Fried, Daniel and Alon, Uri and others},
  journal={arXiv preprint arXiv:2307.13854},
  year={2023}
}

Discussion

1 thread and 0 posts on this page. Anyone may reply; no account is required.

Contribute

Share a reproduction, report an evaluation issue, or suggest a correction to this benchmark overview.

Public problem summary

Catalogue metadata only; public problem statistics are not available.

Public problems
—
Formats
—
Problems with images
—
Answer availability
—
Configs
—
Splits
—
Awaiting a first discussion
—
Problems with a discussion
—
Public contributions
—