{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"93677e33-8de7-5072-82b7-26047c05b546","task_key":"all--test--93677e33-8de7-5072-82b7-26047c05b546","task_revision_id":"1","upstream_id":"","short_description":"A city resident commuted to work each day by subway. After purchasing her subway…","config":"all","split":"test","body":"{\"choices\":\"['prevail, because a special duty of care was owed him.', 'prevail, because the sudden stop caused the resident to fall onto him.', 'not prevail, if the operator of the subway was not negligent in making the stop.', 'not prevail, if the resident was negligent in not holding onto the hand rail.']\",\"question\":\"A city resident commuted to work each day by subway. After purchasing her subway ticket, the resident entered the subway for her ride downtown to where she was employed. Looking for a seat, the resident was walking toward the rear of the subway car when it came to an abrupt halt. As a result of the sudden stop, the resident fell onto a commuter, injuring him. If the commuter asserts a claim against the subway company to recover for his personal injuries, the commuter will probably\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}