{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"8653eb5d-387d-5bb4-866b-e568fccd46f7","task_key":"all--test--8653eb5d-387d-5bb4-866b-e568fccd46f7","task_revision_id":"1","upstream_id":"","short_description":"In which of the following situations would the actions of the defendant(s)…","config":"all","split":"test","body":"{\"choices\":\"['A defendant, seeking shelter from a severe snowstorm, broke into a house. When he awoke the next morning, the defendant noticed a transistor radio, which he put into his pocket before fleeing the house.', 'A defendant was walking past a home when he saw the homeowner sitting in a chair near the window. Angered at the homeowner for stealing his former girlfriend, the defendant picked up a large brick that was lying nearby and hurled it through the window, intending to kill the homeowner.', \\\"Late one afternoon, two defendants were walking down the street when they noticed an open window at a doctor's office. The defendants crawled through the window, broke open his medicine cabinet, and stole various drugs.\\\", 'Late one evening as a guest was registering ata hotel, a defendant noticed that the guest wasa guy who owed him money from a bet. Afterthe guest had retired to his room, the defendantbroke into his room and threatened him regarding the money he was owed.']\",\"question\":\"In which of the following situations would the actions of the defendant(s) constitute a burglary?\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}