{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"37fc4f70-2280-54a1-90c7-dd4a0e9e157f","task_key":"all--test--37fc4f70-2280-54a1-90c7-dd4a0e9e157f","task_revision_id":"1","upstream_id":"","short_description":"The fallacy of extension consists of","config":"all","split":"test","body":"{\"choices\":\"['asking a question that includes either an unproven assumption or more than one question, thus making a straightforward yes or no answer meaningless.', \\\"arguing something is flawed because it doesn't do what it was never intended to do.\\\", \\\"arguing about something when you don't know what you're talking about\\\", 'including more than one claim in the proposition and treating proof for one claim as proof for all the claims.']\",\"question\":\"The fallacy of extension consists of\",\"subject\":\"logical_fallacies\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}