{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"62e6dae3-cd89-5cba-a07e-8614823ed386","task_key":"default--test--1287","task_revision_id":"1","upstream_id":"1287","short_description":"Which of the following criticisms of Llewellyn's distinction between the grand…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['The distinction is too simplistic to capture the complexity of legal reasoning.', 'It overemphasizes the role of judges while ignoring the role of juries.', 'Judges are appointed to interpret the law, not to make it.', 'It is misleading to pigeon-hole judges in this way.', 'Legal reasoning is always grand.', 'It does not consider the role of precedent in legal reasoning.', \\\"Llewellyn's distinction lacks empirical evidence.\\\", 'The distinction does not take into account the influence of personal biases.', 'There is no distinction between the two forms of legal reasoning.', 'Judicial reasoning is always formal.']\",\"question\":\"Which of the following criticisms of Llewellyn's distinction between the grand and formal styles of legal reasoning is the most compelling?\",\"src\":\"ori_mmlu-jurisprudence\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}