{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"3c773120-f86a-5d2f-97bb-b6d154f7996a","task_key":"default--test--4851","task_revision_id":"1","upstream_id":"4851","short_description":"This question refers to the following information.","config":"default","split":"test","body":"{\"category\":\"history\",\"options\":\"['In ancient Rome, religious worship was solely a matter of personal choice and preference.', 'In ancient Rome, religious worship was seen as a moral obligation by the citizens.', 'In ancient Rome, religious worship was heavily influenced by Greek mythology.', 'In ancient Rome, religious worship was the source of much social tension and turmoil.', 'In ancient Rome, religious worship was homogeneous and highly centralized.', \\\"In ancient Rome, religious worship was decentralized and tended to vary with one's social position.\\\", 'In ancient Rome, religious worship was strictly enforced by the state.', 'In ancient Rome, religious worship was considered to be a matter of political importance.', 'In ancient Rome, religious worship was revolutionized by the introduction of Christianity.']\",\"question\":\"This question refers to the following information.\\nRead the the following quotation to answer questions.\\nThe various modes of worship which prevailed in the Roman world were all considered by the people as equally true; by the philosopher as equally false; and by the magistrate as equally useful.\\nEdward Gibbon, The Decline and Fall of the Roman Empire, 1776–1788\\nGibbon's interpretation of the state of religious worship in ancient Rome could be summarized as\",\"src\":\"ori_mmlu-high_school_european_history\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}