{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"4876a7ba-70ad-5c02-b6bf-7cbc94e2582f","task_key":"default--test--4902","task_revision_id":"1","upstream_id":"4902","short_description":"Research on C3 and C4 pathways indicates that changes in plant communities were…","config":"default","split":"test","body":"{\"category\":\"history\",\"options\":\"['meat to fruits and vegetables.', 'species focused on grasses to fruits and vegetables.', 'nuts and fruits to fish.', 'nuts and fruits to species more focused on grasses.', 'species focused on grasses to meat.', 'fruits and vegetables to fish.', 'species focused on grasses to nuts and fruits.', 'fruits and vegetables to species more focused on grasses.', 'meat to species more focused on grasses.', 'fish to fruits and vegetables.']\",\"question\":\"Research on C3 and C4 pathways indicates that changes in plant communities were associated with a transition in the diets of ancient hominids from:\",\"src\":\"ori_mmlu-prehistory\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}