benchmarks.wiki

benchmarks.wiki / Public workspace

tau-bench

Agents

tau-bench places an agent in a customer-service conversation with a simulated user and a domain API. It scores the resulting database state against the rules the agent was given, not the transcript.

tau-bench

Official source

Download answersVisit the official benchmarkWhere the answers areStart a benchmark discussion

History

Sierra researchers introduced tau-bench in 2024 to evaluate agents that interact with users and tools while following domain policies. The motivating problem was reliable completion of customer-service tasks across multiple conversational turns. The official repository recorded in this catalog was created in June 2024.

How it was built

Each task combines a simulated user, a domain policy and an API through which the agent can inspect or change the environment. The agent works through a conversation and tool calls, and evaluation compares the resulting database state with the task's expected outcome. This produces an executable interaction task rather than a static question with a reference sentence.

At a glance

Size
Not imported here; see the official distribution for the current size.
License
MIT (declared on the source)
Released
Repository created on GitHub 2024-06
Maintainer
sierra-research (source organisation)
Answer access
Tasks are executed against a simulated user and a live tool API rather than answered from text, so there is no problem statement to republish.

What a problem looks like

A conversational task with a user simulator, a set of tools and a domain policy the agent must respect.

How it is scored

Tasks are executed against a simulated user and a live tool API rather than answered from text, so there is no problem statement to republish.

Why it is hard

Success depends on multi-turn dialogue, tool use and rule compliance together; the score is the final database state, not the transcript.

Limitations and criticism

The user simulator and tool environment are part of the evaluation, so changes to either can change the observed success rate. Matching the expected database state establishes an outcome but does not by itself establish that every conversational statement was correct or every intermediate action appropriate. Customer-service policies and APIs define a bounded domain, limiting the conclusions that can be drawn about general autonomous work.

Cite

@misc{yao2024tau,
      title={$\tau$-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains}, 
      author={Shunyu Yao and Noah Shinn and Pedram Razavi and Karthik Narasimhan},
      year={2024},
      eprint={2406.12045},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2406.12045}, 
}
@misc{barres2025tau2,
      title={$\tau^2$-Bench: Evaluating Conversational Agents in a Dual-Control Environment}, 
      author={Victor Barres and Honghua Dong and Soham Ray and Xujie Si and Karthik Narasimhan},
      year={2025},
      eprint={2506.07982},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2506.07982}, 
}

Discussion

1 thread and 0 posts on this page. Anyone may reply; no account is required.

Contribute

Share a reproduction, report an evaluation issue, or suggest a correction to this benchmark overview.

Public problem summary

Catalogue metadata only; public problem statistics are not available.

Public problems
—
Formats
—
Problems with images
—
Answer availability
—
Configs
—
Splits
—
Awaiting a first discussion
—
Problems with a discussion
—
Public contributions
—