{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"c13d7b37-b1ae-5394-9f14-02530cf40ece","task_key":"all--test--c13d7b37-b1ae-5394-9f14-02530cf40ece","task_revision_id":"1","upstream_id":"","short_description":"One of the difficulties in 'operationalizing' concepts like social class is…","config":"all","split":"test","body":"{\"choices\":\"['definitions and indicators can vary, making valid comparisons problematic', 'there are no reliable indicators of such widely contested ideas', 'it takes all the fun out of armchair theorising', 'it has little use for applied, empirical research about the topic']\",\"question\":\"One of the difficulties in 'operationalizing' concepts like social class is that:\",\"subject\":\"sociology\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}