{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"da8f7c4c-d160-5dab-9d9c-70947e08506c","task_key":"all--test--da8f7c4c-d160-5dab-9d9c-70947e08506c","task_revision_id":"1","upstream_id":"","short_description":"Science fiction: readers claim to either love it or loathe it; either they avoid…","config":"all","split":"test","body":"{\"choices\":\"['suggest that science fiction is not a serious literary genre', \\\"indicate the depth of certain readers' feelings about science fiction\\\", 'explain why some readers consider science fiction to be dangerous', 'contrast the characteristics of science fiction with those of other literary genres']\",\"question\":\"Science fiction: readers claim to either love it or loathe it; either they avoid it like poison or they devour favorite works and authors like chocolate addicts gulping down fudge truffles. The author of the passage compares certain readers with \\\"chocolate addicts\\\" primarily in order to\",\"subject\":\"miscellaneous\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}