{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"a9de5b49-c10e-5b32-8a49-5945b64337a8","task_key":"default--test--4778","task_revision_id":"1","upstream_id":"4778","short_description":"This question refers to the following information.","config":"default","split":"test","body":"{\"category\":\"history\",\"options\":\"['The population declined as a result of imported disease.', 'The population declined due to the loss of traditional agriculture.', 'The population declined due to the increased competition for resources.', 'The population declined because of famine caused by the mass export of crops.', 'The population increased due to the new trading opportunities.', 'The population increased due to the increased diversity of crops.', 'The population declined due to conflicts over the new resources.']\",\"question\":\"This question refers to the following information.\\nThe invaders had brought in wheat and other Eurasian and African grains; peach, pear, orange, and lemon trees; chick-peas, grape vines, melons, onions, radishes, and much more. A Spanish nobleman come to America could require his Indians to furnish his table with the fruits of his ancestors.\\n—Alfred Crosby, historian, 1972\\nWhat was the impact of this process on Europe?\",\"src\":\"ori_mmlu-high_school_world_history\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}