{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"520cd7df-99a4-5dc2-af69-58fe9fd41371","task_key":"all--test--520cd7df-99a4-5dc2-af69-58fe9fd41371","task_revision_id":"1","upstream_id":"","short_description":"At the birthday party of your best friend, you see Skylar help himself to a…","config":"all","split":"test","body":"{\"choices\":\"['the marginal benefit of the second piece of cake is less than the marginal cost.', 'the total benefit received from eating cake is falling.', 'the ratio of marginal benefit over marginal cost is less than one.', 'the marginal benefit of the second piece of cake is greater than the marginal cost.']\",\"question\":\"At the birthday party of your best friend, you see Skylar help himself to a second piece of cake. For this individual, it must be the case that\",\"subject\":\"high_school_microeconomics\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}