Discussion
0 posts
No posts in this thread yet.
benchmarks.wiki / Public workspace
OSWorld
Agents / Multimodal
OSWorld runs an agent inside a real desktop operating system. It scores whether the requested task was actually accomplished, across applications and file-system state.
osworld
History
The xlang-ai research group introduced OSWorld in 2024 to evaluate agents performing tasks in real desktop operating systems. It addressed the gap between answering questions about an interface and actually carrying out work across applications and files. The October 2023 date recorded for this source is the GitHub repository creation date, rather than the date of the benchmark paper.
How it was built
OSWorld tasks pair a natural-language request with an initial desktop state established by a setup script. An agent operates inside a virtual machine, and an execution-based checker examines whether the requested outcome was achieved. This makes the benchmark a collection of executable environments and task-specific checks, rather than a static set of questions with textual reference answers.
At a glance
What a problem looks like
A natural-language task in a real desktop environment, with a setup script and an execution-based checker.
How it is scored
Every task is executed in a virtual machine and graded by an execution script; there is no text corpus to republish.
Reported scores
Evaluation conditions differ between rows, so this is not a ranking. Rows marked vendor are reports by the model’s own organization.
| Model | Score | Metric | Setup | Reported by | Source |
|---|---|---|---|---|---|
| GPT-4 | 12.24 | Success Rate (%) | A11y tree; overall Ubuntu tasks; max 15 steps; temperature 1.0; top-p 0.9. | paper | Table 5, Overall column; Section 4.1 |
| GPT-4V | 5.26 | Success Rate (%) | Screenshot; overall Ubuntu tasks; max 15 steps; temperature 1.0; top-p 0.9. | paper | Table 5, Overall column; Section 4.1 |
| Claude-3-Opus | 2.42 | Success Rate (%) | Screenshot; overall Ubuntu tasks; max 15 steps; temperature 1.0; top-p 0.9. | paper | Table 5, Overall column; Section 4.1 |
Why it is hard
The agent has to operate real applications through the screen, and partial progress scores nothing.
Limitations and criticism
Success depends on the agent harness as well as the underlying model, including the observation format, action interface and allowed number of steps. Application versions and environment state can also affect whether an interaction succeeds. Execution-based checking captures the conditions encoded by each checker, rather than every aspect of the quality of the finished work. Results using accessibility trees and results using screenshots therefore describe materially different settings.
How much room is left
The paper results recorded here report 12.24 percent success for GPT-4 with accessibility-tree observations and 5.26 percent for GPT-4V with screenshots, both under a 15-step limit on Ubuntu tasks. These historical results leave substantial headroom in those configurations. They do not establish current best performance, and the different observation formats prevent a clean model-only comparison.
Reference links
Cite
@misc{OSWorld,
title={OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments},
author={Tianbao Xie and Danyang Zhang and Jixuan Chen and Xiaochuan Li and Siheng Zhao and Ruisheng Cao and Toh Jing Hua and Zhoujun Cheng and Dongchan Shin and Fangyu Lei and Yitao Liu and Yiheng Xu and Shuyan Zhou and Silvio Savarese and Caiming Xiong and Victor Zhong and Tao Yu},
year={2024},
eprint={2404.07972},
archivePrefix={arXiv},
primaryClass={cs.AI}
}Discussion
1 thread and 0 posts on this page. Anyone may reply; no account is required.
Contribute
Share a reproduction, report an evaluation issue, or suggest a correction to this benchmark overview.
Public problem summary
Catalogue metadata only; public problem statistics are not available.
- Public problems
- —
- Formats
- —
- Problems with images
- —
- Answer availability
- —
- Configs
- —
- Splits
- —
- Awaiting a first discussion
- —
- Problems with a discussion
- —
- Public contributions
- —