{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"4a5c76fe-194f-59cb-8358-8b8211b4fef3","task_key":"all--test--4a5c76fe-194f-59cb-8358-8b8211b4fef3","task_revision_id":"1","upstream_id":"","short_description":"A seat in a movie theater collapsed, causing the customer sitting in that seat…","config":"all","split":"test","body":"{\"choices\":\"['recover, because the theater was under an absolute duty of care to make the premises safe for the protection of its customers.', 'recover, only if the theater had prior knowledge of the dangerous condition of the seat.', 'not recover, because the theater was under no obligation to inspect the premises to discover unknown dangers.', 'not recover, unless the theater failed to make a reasonable inspection of the seats for the safety of its customers.']\",\"question\":\"A seat in a movie theater collapsed, causing the customer sitting in that seat to fall to the ground and suffer a back injury. In a personal injury action by the customer against the theater, the customer will most likely\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}