{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"e76cfb09-c758-5d54-9119-0389d880b2b0","task_key":"all--test--e76cfb09-c758-5d54-9119-0389d880b2b0","task_revision_id":"1","upstream_id":"","short_description":"Philosophy is concerned primarily with identifying beliefs about human existence…","config":"all","split":"test","body":"{\"choices\":\"['why should we bother? and what are the consequences of our believing one thing over another?', 'what do you mean? and how do you know?', \\\"who really believes X? and how can we explain differences in people's beliefs?\\\", 'how do philosophers argue? and are their differences important?']\",\"question\":\"Philosophy is concerned primarily with identifying beliefs about human existence and evaluating arguments that support those beliefs.  These activities can be summarized in two questions that drive philosophical investigations:\",\"subject\":\"philosophy\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}