{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"3480f466-afa4-5c86-9756-f1c04ce48b9c","task_key":"all--test--3480f466-afa4-5c86-9756-f1c04ce48b9c","task_revision_id":"1","upstream_id":"","short_description":"Identify the conclusion of the following argument. Utilitarian theory is…","config":"all","split":"test","body":"{\"choices\":\"['Utilitarian theory is embarrassed by the possibility of utility monsters.', 'A utility monster gets enormous gains in utility.', 'A utility monster gets gains in utility greater than the utility lost by the sacrifices of others.', 'The sacrifices of even large numbers of other people are outweighed by the gains of the utility monster.']\",\"question\":\"Identify the conclusion of the following argument. Utilitarian theory is embarrassed by the possibility of utility monsters. A utility monster gets enormous gains in utility, greater than the utility lost by the sacrifices of others. The sacrifices of even large numbers of other people are outweighed by the gains of the utility monster. The theory seems to require that we all be sacrificed in the monste's maw in order to increase total utility. (Robert Nozick, Anarchy, State, and Utopia)\",\"subject\":\"formal_logic\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}