{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"15f9ceba-5a3e-5225-82c3-ee58a91149ce","task_key":"all--test--15f9ceba-5a3e-5225-82c3-ee58a91149ce","task_revision_id":"1","upstream_id":"","short_description":"_A priori_ is a fallacy that involves","config":"all","split":"test","body":"{\"choices\":\"['asking a question that includes either an unproven assumption or more than one question, thus making a straightforward yes or no answer meaningless.', \\\"arguing something is inferior just because it doesn't do something it was never intended to do.\\\", 'arguing that a claim must be true because there is no evidence that it is false.', 'reasoning that determines the conclusion one wants first, then accepts only evidence supporting that conclusion, or interprets all evidence as support for that conclusion.']\",\"question\":\"_A priori_ is a fallacy that involves\",\"subject\":\"logical_fallacies\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}