{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"ca5d234c-2012-50ad-9ea8-e8d669201af3","task_key":"default--test--6138","task_revision_id":"1","upstream_id":"6138","short_description":"Which of the following is thought to be implicated in the development of…","config":"default","split":"test","body":"{\"category\":\"health\",\"options\":\"['A buildup of carbon dioxide.', 'An excess of antioxidants.', 'An excess of lactic acid.', 'An accumulation of inorganic phosphate.', 'Dehydration of the muscles.', 'A deficiency of calcium.', 'A lack of glycogen in the muscles.', 'Development of hyperosmolality in the muscles.', 'A lack of potassium.', 'An excess of sodium.']\",\"question\":\"Which of the following is thought to be implicated in the development of peripheral muscle fatigue during multiple sprint activities?\",\"src\":\"ori_mmlu-college_medicine\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}