{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"669c28f8-b1d2-5be5-96c3-289f8b348ea3","task_key":"all--test--669c28f8-b1d2-5be5-96c3-289f8b348ea3","task_revision_id":"1","upstream_id":"","short_description":"A preacher and his posse make a habit of showing up at funerals of soldiers…","config":"all","split":"test","body":"{\"choices\":\"['The father, as a funeral is a private affair.', 'The preacher, because it involves a matter of public interest.', 'The father, because time, place, and manner restrictions should have been in place.', 'The father, because the First Amendment does not include protection for vicious verbal assaults such as this.']\",\"question\":\"A preacher and his posse make a habit of showing up at funerals of soldiers suspected to be gay, carrying obnoxious signs and yelling out hurtful epithets. Finally, the parents of one dead soldier has enough and sues the preacher, and the case goes all the way to the Supreme Court. Would the Court likely find in favor of the preacher or the father?\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}