benchmarks.wiki

benchmarks.wiki / Public workspace

Arena-Hard-Auto

Instruction following

Arena-Hard-Auto selects 500 prompts from 200,000 real LMArena user conversations. Answers are scored automatically with a strong model as judge, which is reported to correlate closely with human preference results.

arena-hard

Official source

Download answersVisit the official benchmarkWhere the answers areStart a benchmark discussion

History

The LMArena team developed Arena-Hard-Auto to evaluate responses to difficult prompts drawn from real user conversations using automated preference judgements. The source distribution recorded here is the v0.1 dataset, published under lmarena-ai on Hugging Face in September 2024. Its purpose is to make preference-based comparison more repeatable without collecting fresh human votes for every pair of models.

How it was built

Arena-Hard-Auto selects 500 prompts from a pool of 200,000 real LMArena user conversations. The source associates prompts with topic clusters and evaluates model responses against a baseline response. There is no single reference answer: a judge model compares the two responses, so the evaluation depends on the chosen judge and comparison protocol.

At a glance

Size
500 prompts in the source; none imported.
License
apache-2.0 (declared on the Hugging Face dataset card)
Released
Dataset published on Hugging Face 2024-09
Maintainer
lmarena-ai (source organisation)
Answer access
No problem text is imported, so there is no answer storage for this entry. The source distributes the answers under its own terms.

What a problem looks like

An open-ended user prompt with its topic cluster; answers are compared against a baseline model.

How it is scored

There is no reference answer — a judge model compares two responses, so there is nothing to store as an answer.

Why it is hard

The prompts were chosen for being the ones real users found hard, and they have no single correct answer, so the score depends on judgement quality as much as on the response.

Limitations and criticism

A model judge measures a proxy for human preference, rather than independently verifying every factual or technical claim in an answer. Response style and length can affect preference judgements, and a change in the judge or baseline changes the comparison being made. The selected difficult prompts also do not represent the frequency of tasks in everyday use. Scores from different Arena-Hard versions or judging configurations therefore need separate interpretation.

Cite

@article{li2024arenahard,
  title={From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline},
  author={Tianle Li and Wei-Lin Chiang and Evan Frick and Lisa Dunlap and Tianhao Wu and Banghua Zhu and Joseph E. Gonzalez and Ion Stoica},
  journal={arXiv preprint arXiv:2406.11939},
  year={2024}
}

Discussion

1 thread and 0 posts on this page. Anyone may reply; no account is required.

Contribute

Share a reproduction, report an evaluation issue, or suggest a correction to this benchmark overview.

Public problem summary

Catalogue metadata only; public problem statistics are not available.

Public problems
—
Formats
—
Problems with images
—
Answer availability
—
Configs
—
Splits
—
Awaiting a first discussion
—
Problems with a discussion
—
Public contributions
—