benchmarks.wiki

benchmarks.wiki / Public workspace

AgentHarm

Safety / Agents

AgentHarm measures how far a tool-using agent will go in carrying out a harmful request. It pairs 110 harmful behaviours with 110 matched benign ones, so over-refusal is measured alongside compliance.

agentharm

Official source

Download answersVisit the official benchmarkWhere the answers areStart a benchmark discussion

History

AgentHarm was introduced in 2024 to evaluate whether tool-using language model agents would carry out harmful requests over multiple steps. The AI Safety Institute distributes the dataset, whose Hugging Face publication is recorded as October 2024. Its design pairs harmful tasks with benign counterparts to distinguish resistance to harmful requests from indiscriminate refusal.

How it was built

The source describes 110 harmful behaviours and 110 matched benign behaviours, presented as multi-step tasks using synthetic tools. Each behaviour has a grading function that checks the agent transcript. The pairing supports assessment of harmful completion alongside performance on related legitimate tasks.

At a glance

Metric
harm score and refusal rate, with the benign pairs measuring over-refusal
Size
110 harmful and 110 benign behaviours in the source; none imported.
License
other (declared on the Hugging Face dataset card)
Released
Dataset published on Hugging Face 2024-10
Maintainer
ai-safety-institute (source organisation)
Answer access
No problem text is imported, so there is no answer storage for this entry. The source distributes the answers under its own terms.

What a problem looks like

Multi-step agent tasks with a set of synthetic tools, each paired with a benign counterpart.

How it is scored

Scored by a grading function per behaviour against the agent transcript; nothing is imported here.

Metric: harm score and refusal rate, with the benign pairs measuring over-refusal

Why it is hard

A single refusal is not enough: the agent has to keep refusing across a multi-step trajectory, and a jailbreak that survives one step often carries through the rest.

Limitations and criticism

Synthetic tools provide a controlled setting, but successful completion does not establish that an agent could cause the corresponding harm in a live environment. Results depend on the tools, attack prompts and trajectory grading used in an evaluation. Refusal rates need to be read alongside benign-task performance, since refusing everything can suppress harmful completion without demonstrating useful discrimination.

Cite

@article{andriushchenko2024agentharm,
  title={AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents},
  author={Maksym Andriushchenko and Alexandra Souly and Mateusz Dziemian and Derek Duenas and Maxwell Lin and Justin Wang and Dan Hendrycks and Andy Zou and Zico Kolter and Matt Fredrikson and Eric Winsor and Jerome Wynne and Yarin Gal and Xander Davies},
  journal={arXiv preprint arXiv:2410.09024},
  year={2024}
}

Discussion

1 thread and 0 posts on this page. Anyone may reply; no account is required.

Contribute

Share a reproduction, report an evaluation issue, or suggest a correction to this benchmark overview.

Public problem summary

Catalogue metadata only; public problem statistics are not available.

Public problems
—
Formats
—
Problems with images
—
Answer availability
—
Configs
—
Splits
—
Awaiting a first discussion
—
Problems with a discussion
—
Public contributions
—