{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"4b4ed82a-9adf-59b6-8653-ef20e0dd221d","task_key":"all--test--4b4ed82a-9adf-59b6-8653-ef20e0dd221d","task_revision_id":"1","upstream_id":"","short_description":"While on a camping trip, a defendant became intoxicated and decided to take a…","config":"all","split":"test","body":"{\"choices\":\"['he was drunk when he broke into the cabin.', 'the crime was not completed until the morning hours.', 'he did not have the requisite intent at the time of the breaking and entering.', 'the cabin was empty when he entered it.']\",\"question\":\"While on a camping trip, a defendant became intoxicated and decided to take a walk late at night. He was so drunk he could not find his way back to the campsite. He did, however, come across a secluded mountain cabin, in which he decided to take shelter for the night. Since the door to the cabin was locked, he broke a window and entered the structure. Once inside, he fell asleep on the sofa. When he awoke the next morning he was hungry. The defendant then found some food in the refrigerator, which he cooked and ate. Before leaving the cabin, the defendant looked around to see if there was anything worth stealing. He opened the door to the bedroom and found a gold watch on the nightstand. He placed the watch in his pocket and left the cabin. The defendant was later arrested and charged with burglary. The defendant's strongest defense would be that\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}