{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"a5a7dcc3-f0d1-5fff-b16d-d9a5a888663e","task_key":"all--test--a5a7dcc3-f0d1-5fff-b16d-d9a5a888663e","task_revision_id":"1","upstream_id":"","short_description":"If a consumer is not required to pay a monetary price for each cookie she…","config":"all","split":"test","body":"{\"choices\":\"['the total utility from eating cookies is equal to zero.', 'the substitution effect outweighs the income effect from eating cookies.', 'the ratio of marginal utility divided by total utility is equal to one.', 'the marginal utility from eating the last cookie is zero.']\",\"question\":\"If a consumer is not required to pay a monetary price for each cookie she consumes, the consumer will stop eating cookies when\",\"subject\":\"high_school_microeconomics\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}