{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"d5cbb450-0ca0-5da5-aa53-8694023e6259","task_key":"default--test--9545","task_revision_id":"1","upstream_id":"9545","short_description":"In a mixture of hydrogen, oxygen, and nitrogen gases at a given temperature, the…","config":"default","split":"test","body":"{\"category\":\"physics\",\"options\":\"['Nitrogen and hydrogen have the same average speed, but oxygen is slower', 'nitrogen', 'hydrogen', 'The speed depends on the specific temperature and cannot be determined', 'Nitrogen and oxygen have the same average speed, but hydrogen is slower', 'Oxygen and hydrogen have the same average speed, but nitrogen is slower', 'The speed of the gases is determined by their density, not their type', 'All have same average speed', 'The speed of the gases is independent of their type and depends solely on their volume', 'oxygen']\",\"question\":\"In a mixture of hydrogen, oxygen, and nitrogen gases at a given temperature, the fastest molecules on average are those of\",\"src\":\"ori_mmlu-conceptual_physics\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}