{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"5a30d53f-0859-53ad-b876-52268a000017","task_key":"default--test--1849","task_revision_id":"1","upstream_id":"1849","short_description":"American realists argued that formalism devalued the authority of the judge to…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['Judicial decisions are less important than legislation.', 'The structure of legal reasoning is flawed.', 'Judges are not allowed to interpret the law, only apply it.', 'Formalism does not allow for moral considerations in the judiciary process.', 'Formalism treats legal reasoning as syllogistic reasoning.', 'Formalism undervalues the influence of societal changes on law.', 'Judges generally base their decisions on formal logic.', 'It assumes all laws are written with perfect clarity.', 'Formalism makes the law rigid and inflexible.', \\\"Judges don't have the necessary training to understand formal logic.\\\"]\",\"question\":\"American realists argued that formalism devalued the authority of the judge to make law, because:\",\"src\":\"ori_mmlu-jurisprudence\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}