{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"afd1cfa8-88ba-5f99-9127-a02533511b17","task_key":"all--test--afd1cfa8-88ba-5f99-9127-a02533511b17","task_revision_id":"1","upstream_id":"","short_description":"A college student was walking home from a party. He had been drinking alcohol at…","config":"all","split":"test","body":"{\"choices\":\"['prevail, unless the police made a valid arrest.', 'prevail, if he consented to the confinement because he was intoxicated.', 'not prevail, because police cannot be held liable for false imprisonment.', 'not prevail, because the student suffered no harm.']\",\"question\":\"A college student was walking home from a party. He had been drinking alcohol at the party and was very intoxicated. Local police officers stopped the student and questioned him. When he refused to answer any of their questions, the police put the student in the back seat of a police car. The police left the student in the back seat of the police car for approximately one hour, and then released him. If the student asserts a claim against the police for false imprisonment, he will most likely\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}