{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"cbe08e68-32d0-5cd9-9aa7-e0e68f0c510e","task_key":"default--test--1334","task_revision_id":"1","upstream_id":"1334","short_description":"A 12-year-old boy decided to play a practical joke. Knowing that his parents…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['the boy loses, because this \\\"practical joke\\\" posed a risk of severe harm and, therefore, the boy had a duty to exercise as much care as a reasonable adult.', \\\"the boy wins, because a child's conduct cannot be negligent.\\\", 'the boy loses, because he did not act with the amount of care that one would expect from a 12-year-old child.', 'the boy wins, because he could not foresee that a guest would mistakenly enter the guest bedroom.', 'the boy wins, because he did not expect anyone to enter the guest bedroom.', 'the boy loses, as he left the house without removing the trap, knowing his parents were having guests over.', 'the boy wins, because the guest should have been more careful in an unfamiliar house.', 'the boy wins, because the guest unreasonably assumed the risk of entering an unfamiliar room rather than asking for directions to the bathroom.', 'the boy loses, because he intentionally set a trap that could potentially cause harm.']\",\"question\":\"A 12-year-old boy decided to play a practical joke. Knowing that his parents were giving a dinner party that evening, the boy filled a bucket with ice-cold water and balanced it on the partially open door of the guest bedroom, thinking that his father would take the guests' coats there to put them on the bed. The boy then went off to play at a friend's house. The boy's father decided to keep all the guests' coats in the hall closet instead of putting them in the guest bedroom. Later that evening, one of the guests mistakenly wandered into the guest bedroom while searching for a bathroom. As the guest opened the door, the bucket fell and hit his head, causing a severe cut that required a dozen stitches. If the guest sues the boy for negligence, the most likely result is\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}