{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"027b8845-5b78-598e-8e1a-4b5336453e08","task_key":"all--test--027b8845-5b78-598e-8e1a-4b5336453e08","task_revision_id":"1","upstream_id":"","short_description":"Information about what the students in Mr. Paulson’s class and in Ms. Hugo’s…","config":"all","split":"test","body":"{\"choices\":\"['whether the lunches were eaten on the same day', 'the total number of students in each teacher’s class', 'nothing because the portions each class ate were the same', 'the total number of pizza slices and hamburgers eaten that day']\",\"question\":\"Information about what the students in Mr. Paulson’s class and in Ms. Hugo’s class ate for lunch is listed below. In Mr. Paulson’s class, 0.5 of the students ate pizza for lunch. In Ms. Hugo’s class, 0.5 of the students ate hamburgers for lunch. What information, if any, is needed to correctly compare the 0.5 of Mr. Paulson’s class that ate pizza to the 0.5 of Ms. Hugo’s class that ate hamburgers?\",\"subject\":\"elementary_mathematics\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}