benchmarks.wiki

benchmarks.wiki / Public workspace

MT-Bench

Instruction following

MT-Bench is a set of 80 two-turn questions across eight categories, published with the study that established LLM-as-a-judge evaluation. It looks at multi-turn consistency and instruction following.

mt-bench

Official source

Download answersVisit the official benchmarkWhere the answers areStart a benchmark discussion

History

Researchers in the LMSYS group introduced MT-Bench in 2023 as part of their study of language models as judges of assistant responses. They designed it to examine instruction following and conversation across two turns, including dependencies that a single-turn test cannot capture. The HuggingFaceH4 prompt distribution was published in July 2023.

How it was built

MT-Bench contains 80 two-turn questions spanning eight categories. The second user turn builds on the first exchange, allowing the evaluation to test whether an assistant follows the conversation as well as the immediate instruction. A judge model assigns scores on a 1-10 scale, with results reported by turn and averaged across categories; most categories do not have a single reference answer.

At a glance

Metric
average judge score over the eight categories, reported per turn
Size
80 questions in the source; none imported.
License
apache-2.0 (declared on the Hugging Face dataset card)
Released
Dataset published on Hugging Face 2023-07
Maintainer
HuggingFaceH4 (source organisation)
Answer access
No problem text is imported, so there is no answer storage for this entry. The source distributes the answers under its own terms.

What a problem looks like

Two successive user turns in one category, the second depending on the answer to the first.

How it is scored

Scored by a judge model on a 1–10 scale; there is no reference answer to store for most categories.

Metric: average judge score over the eight categories, reported per turn

Why it is hard

The second turn depends on the first, so an answer that was locally fine but unhelpful for what follows is penalised where a single-turn benchmark would not notice.

Limitations and criticism

Model-based judging can reward verbosity and exhibit position or self-preference biases, issues examined in the original study. Scores depend on the judge model and judging prompt, and are not interchangeable with objectively verified task accuracy. Eighty short conversations offer limited coverage of extended interaction, while the public prompts can become targets for training or benchmark-specific optimisation.

Cite

@article{zheng2023mtbench,
  title={Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena},
  author={Lianmin Zheng and Wei-Lin Chiang and Ying Sheng and Siyuan Zhuang and Zhanghao Wu and Yonghao Zhuang and Zi Lin and Zhuohan Li and Dacheng Li and Eric P. Xing and Hao Zhang and Joseph E. Gonzalez and Ion Stoica},
  journal={arXiv preprint arXiv:2306.05685},
  year={2023}
}

Discussion

1 thread and 0 posts on this page. Anyone may reply; no account is required.

Contribute

Share a reproduction, report an evaluation issue, or suggest a correction to this benchmark overview.

Public problem summary

Catalogue metadata only; public problem statistics are not available.

Public problems
—
Formats
—
Problems with images
—
Answer availability
—
Configs
—
Splits
—
Awaiting a first discussion
—
Problems with a discussion
—
Public contributions
—