{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"36d010f4-2dcd-51af-b1f2-6b2045650a15","task_key":"default--test--1045","task_revision_id":"1","upstream_id":"1045","short_description":"A man was on a date when he realized that he didn't have enough money to pay the…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['not guilty, because he did not intend to burn down the restaurant.', 'not guilty, because the fire only charred a portion of the ceiling.', 'not guilty, because he did not have the intent to cause harm to any individuals.', \\\"guilty, because the act caused damage to the restaurant's ceiling.\\\", \\\"guilty, because he was reckless as to the restaurant's being damaged by fire.\\\", 'guilty, because he set the fire for the purpose of committing an unlawful act.', 'guilty, because the fire was intentionally started, regardless of the purpose.', 'not guilty, because the fire did not result in significant damage to the structure.', 'not guilty, because he did not directly set fire to the structure itself.', 'guilty, because he set fire with an intent to create a diversion.']\",\"question\":\"A man was on a date when he realized that he didn't have enough money to pay the bill. The man excused himself and went to the men's room. The man decided to set fire to the waste paper in a trash container. He hoped that the fire would serve as a diversion so he and his date could run out of the restaurant without paying. He set fire to the waste paper and then went back to his table. As he anxiously waited, the maitre d' grabbed the man and told him that he had seen what occurred in the bathroom. The maitre d', who had been seated in a bathroom stall, saw the man, through a crack in the door, set the fire. The maitre d' extinguished the fire, which had charred a portion of the ceiling. This jurisdiction defines arson as the malicious burning of any structure. If charged with arson, the man should be found\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}