Discussion
0 posts
No posts in this thread yet.
benchmarks.wiki / Public workspace
MuSR
Reasoning
MuSR poses multistep soft reasoning problems over algorithmically generated narratives. It has three domains — murder mysteries, object placements and team allocation — and this catalogue imports all three domains, 756 problems in total.
musr
Full text: eligible public problems include text, images and discussion.
History
Zayne Sprague and collaborators introduced MuSR in 2023 to study multistep reasoning in narratives where individual clues require commonsense interpretation. The benchmark covered murder mysteries, object placement, and team allocation rather than relying only on explicitly stated logical rules. The catalogue lists TAUR-Lab as the source maintainer and records the Hugging Face dataset publication in May 2024.
How it was built
MuSR uses algorithmically generated narratives across three domains: murder mysteries, object placement and team allocation. Each problem presents a narrative, a question and candidate answers, while the reference data identifies the correct option. This catalogue records 250 problems in the murder-mystery selection and imports 200 of them, rather than representing all three domains.
At a glance
What a problem looks like
A narrative of several hundred words, a question about it, and the candidate answers as a list. The correct index is not part of the prompt.
How it is scored
The source ships the answer index and the chosen option; this catalogue keeps both in restricted storage and never renders them.
Metric: accuracy over the 250 problems in each domain
Why it is hard
The clues are spread through a long narrative and none of them decides the answer alone. The chain has to be assembled from commonsense inferences that the text never states.
Limitations and criticism
The generated narratives provide a controlled setting but do not establish performance on unrestricted real-world reasoning. Multiple-choice accuracy measures selection of the reference option without directly verifying the reasoning used to reach it. The murder-mystery selection also cannot substitute for an aggregate across object placement and team allocation, whose reasoning demands differ.
Reference links
Cite
@article{sprague2023musr,
title={MuSR: Testing the Limits of Chain-of-thought with Multistep Soft Reasoning},
author={Zayne Sprague and Xi Ye and Kaj Bostrom and Swarat Chaudhuri and Greg Durrett},
journal={arXiv preprint arXiv:2310.16049},
year={2023}
}Discussion
1 thread and 0 posts on this page. Anyone may reply; no account is required.
Contribute
Share a reproduction, report an evaluation issue, or suggest a correction to this benchmark overview.
Tasks
Whole public benchmark.
- Public problems
- 756
- Formats
- text (756)
- Problems with images
- 0
- Answer availability
- Answer published by the source (756)
- Configs
- default (756)
- Splits
- murder_mysteries (250) · object_placements (256) · team_allocation (250)
- Awaiting a first discussion
- 756
- Problems with a discussion
- 0
- Public contributions
- 0
Provenance and terms
Pinned source revisions, newest first:
7c365b439a222150f317764d4f16ae6c96d7d94a· acquired 2026-09-23 · release4eea9fa155227c365b439a222150f317764d4f16ae6c96d7d94a· acquired 2026-09-12 · release2a1b3a07a10a
- Who is the most likely murderer?
Answer published by the source
- Who is the most likely murderer?
Answer published by the source
- Who is the most likely murderer?
Answer published by the source
- Who is the most likely murderer?
Answer published by the source
- Who is the most likely murderer?
Answer published by the source
- Who is the most likely murderer?
Answer published by the source
- Who is the most likely murderer?
Answer published by the source
- Who is the most likely murderer?
Answer published by the source
- Who is the most likely murderer?
Answer published by the source
- Who is the most likely murderer?
Answer published by the source
- Who is the most likely murderer?
Answer published by the source
- Which location is the most likely place Marco would look to find the artifact…
Answer published by the source
- Which location is the most likely place Jenny would look to find the private…
Answer published by the source
- Which location is the most likely place Laura would look to find the logbook…
Answer published by the source
- Which location is the most likely place Laura would look to find the specific…
Answer published by the source
- Which location is the most likely place Francesca would look to find the antique…
Answer published by the source
- Which location is the most likely place Emma would look to find the trowel given…
Answer published by the source
- Which location is the most likely place Mary would look to find the backdrop…
Answer published by the source
- Which location is the most likely place Claire would look to find the…
Answer published by the source
- Which location is the most likely place Matthew would look to find the pen given…
Answer published by the source
- Which location is the most likely place Fred would look to find the secret sauce…
Answer published by the source
- Which location is the most likely place Alice would look to find the dorothy's…
Answer published by the source
- Which location is the most likely place Maria would look to find the notepad…
Answer published by the source
- Which location is the most likely place Rachel would look to find the spare…
Answer published by the source
- Which location is the most likely place Emma would look to find the notebook…
Answer published by the source
- Which location is the most likely place Max would look to find the confidential…
Answer published by the source
- Which location is the most likely place Simon would look to find the book given…
Answer published by the source
- Which location is the most likely place Andrew would look to find the violin…
Answer published by the source
- Which location is the most likely place Grandma would look to find the…
Answer published by the source
- Which location is the most likely place Alan would look to find the vintage lamp…
Answer published by the source
- Which location is the most likely place Nancy would look to find the micro…
Answer published by the source
- Which location is the most likely place John would look to find the helmets…
Answer published by the source
- Given the story, how would you uniquely allocate each person to make sure both…
Answer published by the source
- Given the story, how would you uniquely allocate each person to make sure both…
Answer published by the source
- Given the story, how would you uniquely allocate each person to make sure both…
Answer published by the source
- Given the story, how would you uniquely allocate each person to make sure both…
Answer published by the source
- Given the story, how would you uniquely allocate each person to make sure both…
Answer published by the source
- Given the story, how would you uniquely allocate each person to make sure both…
Answer published by the source
- Given the story, how would you uniquely allocate each person to make sure both…
Answer published by the source
- Given the story, how would you uniquely allocate each person to make sure both…
Answer published by the source
- Given the story, how would you uniquely allocate each person to make sure both…
Answer published by the source
- Given the story, how would you uniquely allocate each person to make sure both…
Answer published by the source
- Given the story, how would you uniquely allocate each person to make sure both…
Answer published by the source
- Given the story, how would you uniquely allocate each person to make sure both…
Answer published by the source
- Given the story, how would you uniquely allocate each person to make sure both…
Answer published by the source
- Given the story, how would you uniquely allocate each person to make sure both…
Answer published by the source
- Given the story, how would you uniquely allocate each person to make sure both…
Answer published by the source
- Given the story, how would you uniquely allocate each person to make sure both…
Answer published by the source
- Given the story, how would you uniquely allocate each person to make sure both…
Answer published by the source
- Given the story, how would you uniquely allocate each person to make sure both…
Answer published by the source