{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"b661726d-92df-5203-8b70-34de6e6d05a1","task_key":"all--test--b661726d-92df-5203-8b70-34de6e6d05a1","task_revision_id":"1","upstream_id":"","short_description":"Wellman's response to the \"egalitarian\" objection to the state's right to…","config":"all","split":"test","body":"{\"choices\":\"['egalitarians are committed to utilitarian principles.', 'Wellman himself is an immigrant.', 'excluding people who wish to pursue interests specific to that state is still potentially harmful.', 'the concept of \\\"exporting justice\\\" is internally contradictory.']\",\"question\":\" Wellman's response to the \\\"egalitarian\\\" objection to the state's right to exclude is that states can fulfill their duties to outsiders without allowing them into their territory. Fine thinks Wellman's response is insufficient because\",\"subject\":\"moral_disputes\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}