{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"e70ecb03-6ec8-503c-9175-97b78654b4b5","task_key":"default--test--1013","task_revision_id":"1","upstream_id":"1013","short_description":"A person was taking a long hike on a trail through a forest in a state park. As…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"[\\\"recover, if the hiker can prove negligence on the hunter's part.\\\", 'recover, if the hunter knew the hiker was there when he turned.', \\\"recover, because the hunter's rifle was pointed directly at him.\\\", 'recover, if the hiker feared for his life.', 'not recover, if the hunter did not intend to shoot the hiker.', 'not recover, unless the hiker can prove the hunter saw him.', 'not recover, if the hiker failed to make his presence known to the hunter.', \\\"recover, because the hunter's decision to carry the gun was voluntary.\\\", 'not recover, unless the hunter intended to scare the hiker.']\",\"question\":\"A person was taking a long hike on a trail through a forest in a state park. As the hiker came around a bend in the trail, he suddenly saw a hunter standing next to the trail, looking in the other direction. The hunter was holding a rifle. The hunter turned, causing the rifle to be pointed at the hiker. The hiker felt sure that he was going to get shot, but the gun did not go off. The hiker asserts a claim for assault against the hunter. In his action, the most likely result is that the hiker will\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}