{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"65e27c6d-c0bb-5509-ac09-20dd8650a153","task_key":"default--test--5051","task_revision_id":"1","upstream_id":"5051","short_description":"This question refers to the following information.","config":"default","split":"test","body":"{\"category\":\"history\",\"options\":\"['that man should not aspire to be whatever he chooses to be', 'that God intended man to strive to achieve', 'that all beings share the same nature', \\\"that God is indifferent to man's happiness\\\", \\\"that humans are God's equals\\\", 'that man was a unique kind of creature', 'that man has limited potential', 'that man is unworthy of admiration', 'that man has no control over his own destiny', 'that man is the least fortunate of all creatures']\",\"question\":\"This question refers to the following information.\\nAt last it seems to me that I have come to understand why man is the most fortunate of all creatures and consequently worthy of all admiration. … The nature of all other beings is limited. … Imagine! The great generosity of God! The happiness of man! To man it is allowed to be whatever he chooses to be!\\nPico della Mirandola, Oration on the Dignity of Man, 1486\\nFrom the passage, one may infer that Pico rejected the notion\",\"src\":\"ori_mmlu-high_school_european_history\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}