{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"10fe242f-b5ee-5b23-be2c-50b7770858a1","task_key":"all--test--10fe242f-b5ee-5b23-be2c-50b7770858a1","task_revision_id":"1","upstream_id":"","short_description":"A hunter sat quietly in a forest, hiding behind some brush and waiting for deer…","config":"all","split":"test","body":"{\"choices\":\"['prevail, because the hunter had no intent to harm the police officer and, therefore, the police officer was not entitled to fire back.', 'prevail, because the hunter had not committed any crime entitling the police officer to use force likely to cause death or serious bodily injury.', 'not prevail, if the police officer reasonably believed that he was under attack.', 'not prevail, because the police officer was not the original aggressor.']\",\"question\":\"A hunter sat quietly in a forest, hiding behind some brush and waiting for deer to pass by him. After hours of waiting and not spotting any deer, the hunter decided to give up and go home. Disappointed that he never even got a chance to take a shot at a deer, the hunter decided to fire a few practice shots before walking back to his car. He aimed his rifle at a small tree at the top of a nearby hill and fired three shots. At that moment, a state police officer happened to be passing nearby. The officer was part of a police team searching the forest for a fugitive who was believed to be armed and dangerous. When the police officer heard the gunshots, he thought he was being attacked. He quickly pulled his pistol out of his holster and fired it toward the brush from which he had seen the muzzle flash from the hunter's rifle. One of the bullets fired by the police officer struck the hunter and seriously wounded him. If the hunter asserts a claim against the police officer to recover damages for his injury, the hunter will\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}