{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"90ceb813-1f7e-58cb-9456-c3b5ee021dfb","task_key":"all--test--90ceb813-1f7e-58cb-9456-c3b5ee021dfb","task_revision_id":"1","upstream_id":"","short_description":"Fine asks us to consider the example of a yoga group that practices in Central…","config":"all","split":"test","body":"{\"choices\":\"['Central Park qualifies as a state.', 'there is a difference between rules of membership and rules about settlement within a physical territory.', 'territorial rights only belong to individuals.', 'Wellman is committed to an absurd theory of group membership.']\",\"question\":\" Fine asks us to consider the example of a yoga group that practices in Central Park and is free to reject prospective members but isn't free to exclude others from making use of Central Park. The point of this example is to show that\",\"subject\":\"moral_disputes\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}