{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"e83ec2af-c568-5730-8472-c9c22b9b4977","task_key":"default--test--9994","task_revision_id":"1","upstream_id":"9994","short_description":"Which of the following lists the ingredients of the solar nebula from highest to…","config":"default","split":"test","body":"{\"category\":\"physics\",\"options\":\"['rocks, metals, light gases (H He), hydrogen compounds (H2O CH4 NH3)', 'light gases (H He) hydrogen compounds (H2O CH4 NH3) metals rocks', 'hydrogen compounds (H2O CH4 NH3) light gases (H He) metals rocks', 'hydrogen compounds (H2O CH4 NH3) light gases (H He) rocks metals', 'metals, hydrogen compounds (H2O CH4 NH3), light gases (H He), rocks', 'metals, rocks, light gases (H He), hydrogen compounds (H2O CH4 NH3)', 'metals, rocks, hydrogen compounds (H2O CH4 NH3), light gases (H He)', 'light gases (H He) hydrogen compounds (H2O CH4 NH3) rocks metals', 'hydrogen compounds (H2O CH4 NH3), metals, rocks, light gases (H He)', 'rocks, hydrogen compounds (H2O CH4 NH3), light gases (H He), metals']\",\"question\":\"Which of the following lists the ingredients of the solar nebula from highest to lowest percentage of mass in the nebula?\",\"src\":\"ori_mmlu-astronomy\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}