benchmarks.wiki

benchmarks.wiki / Public workspace

tau-bench

Agents

tau-bench places an agent in a customer-service conversation with a simulated user and a domain API. It scores the resulting database state against the rules the agent was given, not the transcript.

tau-bench

Overview onlyNot imported on this site

Official source

Overview only: problem text has not been published on this site; the catalogue records the benchmark introduction, category and official source.

Visit the official benchmarkStart a benchmark discussion
Catalogue status
Not imported on this site

At a glance

Size
Not imported here; see the official distribution for the current size.
License
MIT (declared on the source)
Released
Repository created on GitHub 2024-06
Maintainer
sierra-research (source organisation)
Answer access
Tasks are executed against a simulated user and a live tool API rather than answered from text, so there is no problem statement to republish.

What a problem looks like

A conversational task with a user simulator, a set of tools and a domain policy the agent must respect.

How it is scored

Tasks are executed against a simulated user and a live tool API rather than answered from text, so there is no problem statement to republish.

Why it is hard

Success depends on multi-turn dialogue, tool use and rule compliance together; the score is the final database state, not the transcript.

Cite

@misc{yao2024tau,
      title={$\tau$-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains}, 
      author={Shunyu Yao and Noah Shinn and Pedram Razavi and Karthik Narasimhan},
      year={2024},
      eprint={2406.12045},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2406.12045}, 
}
@misc{barres2025tau2,
      title={$\tau^2$-Bench: Evaluating Conversational Agents in a Dual-Control Environment}, 
      author={Victor Barres and Honghua Dong and Soham Ray and Xujie Si and Karthik Narasimhan},
      year={2025},
      eprint={2506.07982},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2506.07982}, 
}

Discussion

Contribute

Share a reproduction, report an evaluation issue, or suggest a correction to this benchmark overview.

Public problem summary

Catalogue metadata only; public problem statistics are not available.

Public problems
—
Formats
—
Problems with images
—
Answer availability
—
Configs
—
Splits
—
Awaiting a first discussion
—
Problems with a discussion
—
Public contributions
—