{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"3d940a8c-551e-5ed3-9ba9-405691968143","task_key":"default--test--4968","task_revision_id":"1","upstream_id":"4968","short_description":"Analysis of the teeth of a hominid species discovered in Malapa Cave in South…","config":"default","split":"test","body":"{\"category\":\"history\",\"options\":\"['meat, fish, and some plants.', 'primarily C3 plants that included barley and oats.', 'large quantities of insects and small mammals.', 'a balanced diet of fruits, vegetables, meat, and fish.', 'primarily C4 plants that included corn and potatoes.', 'C4 grasses that may have included wild wheat.', 'fruits, leaves, and C3 plants.', 'meat derived from carnivorous mammals.', 'nuts, seeds, and roots.', 'primarily aquatic plants and algae.']\",\"question\":\"Analysis of the teeth of a hominid species discovered in Malapa Cave in South Africa, dating just after 2 million years ago, indicates a diet that consisted primarily of:\",\"src\":\"ori_mmlu-prehistory\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}