{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"7299f718-8312-5090-97fd-0d67dcb6a10d","task_key":"all--test--7299f718-8312-5090-97fd-0d67dcb6a10d","task_revision_id":"1","upstream_id":"","short_description":"Tyler is a high school student who is planning on becoming an engineer. In his…","config":"all","split":"test","body":"{\"choices\":\"['He decides that the first test is always harder than the others.', 'He says the teacher graded his exam harder because she doesn’t like him.', 'He says it was due to some home circumstances that won’t be present during the next exam.', 'He critiques his study methods and tries to find out which led to poor returns.']\",\"question\":\"Tyler is a high school student who is planning on becoming an engineer. In his calculus II class sophomore year, he receives an F on his first test. Which of the following responses to this event would indicate that Tyler has a higher likelihood of improving in subsequent exams?\",\"subject\":\"college_medicine\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}