{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"bfd90a16-010d-5b5d-b822-91e8dff14ab7","task_key":"default--test--1277","task_revision_id":"1","upstream_id":"1277","short_description":"A preacher and his posse make a habit of showing up at funerals of soldiers…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"[\\\"The father, because the preacher's actions disturbed the peace of a private event.\\\", 'The preacher, because it involves a matter of public interest.', 'The preacher, because the First Amendment protects even offensive or hateful speech.', 'The father, as a funeral is a private affair.', 'The preacher, because his actions are protected under freedom of speech and religion.', \\\"The father, because the preacher's actions were intentionally disruptive and disrespectful.\\\", 'The preacher, because he has a right to express his religious views regardless of the context.', 'The father, because the First Amendment does not include protection for vicious verbal assaults such as this.', \\\"The father, because the preacher's actions constitute harassment and are therefore not protected by the First Amendment.\\\"]\",\"question\":\"A preacher and his posse make a habit of showing up at funerals of soldiers suspected to be gay, carrying obnoxious signs and yelling out hurtful epithets. Finally, the parents of one dead soldier has enough and sues the preacher, and the case goes all the way to the Supreme Court. Would the Court likely find in favor of the preacher or the father?\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}