{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"d7d2ceba-cf5b-5a87-95cd-e6d8d9bffd4b","task_key":"all--test--d7d2ceba-cf5b-5a87-95cd-e6d8d9bffd4b","task_revision_id":"1","upstream_id":"","short_description":"Whereas the social sciences (e.g., psychology, sociology, economics) ask…","config":"all","split":"test","body":"{\"choices\":\"['how people with different beliefs or backgrounds disagree with one another.', 'what beliefs mean and whether people with different beliefs are justified in having them.', 'the reasons why philosophic questions never have better or worse answers.', 'questions that can be answered better by appealing to scientific experiments.']\",\"question\":\"Whereas the social sciences (e.g., psychology, sociology, economics) ask questions about how people think and act, philosophy is the study of:\",\"subject\":\"philosophy\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}