{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"ffac1cf0-978f-520e-b3e9-d54ca602292f","task_key":"all--test--ffac1cf0-978f-520e-b3e9-d54ca602292f","task_revision_id":"1","upstream_id":"","short_description":"A person was taking a long hike on a trail through a forest in a state park. As…","config":"all","split":"test","body":"{\"choices\":\"[\\\"recover, because the hunter's rifle was pointed directly at him.\\\", \\\"recover, because the hunter's decision to carry the gun was voluntary.\\\", 'not recover, unless the hunter intended to scare the hiker.', 'not recover, if the hunter did not intend to shoot the hiker.']\",\"question\":\"A person was taking a long hike on a trail through a forest in a state park. As the hiker came around a bend in the trail, he suddenly saw a hunter standing next to the trail, looking in the other direction. The hunter was holding a rifle. The hunter turned, causing the rifle to be pointed at the hiker. The hiker felt sure that he was going to get shot, but the gun did not go off. The hiker asserts a claim for assault against the hunter. In his action, the most likely result is that the hiker will\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}