{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"dc2e0b34-992f-5cd4-8992-4389a4ddbc99","task_key":"default--test--3252","task_revision_id":"1","upstream_id":"3252","short_description":"When did the first mammals appear? Compare the monotremesand marsupials.","config":"default","split":"test","body":"{\"category\":\"biology\",\"options\":\"['The first mammals originated during the Carboniferous Period of the Paleozoic Era.', 'The first mammals emerged in the Silurian Period, which is part of the Paleozoic Era.', 'The first mammals arose in the Cenozoic Era.', 'The first mammals arose shortly after the end of the Cretaceous Period, in the early Paleogene of the Cenozoic Era.', 'The first mammals emerged in the late Cretaceous Period of the Mesozoic Era.', 'The first mammals arose in the earliest period (the Triassic) of the Mesozoic Era which dates to some 230 million years ago.', 'The first mammals arose in the Paleozoic Era.', 'The first mammals arose in the Jurassic Period.', 'The first mammals appeared at the beginning of the Permian Period.', 'The first mammals arose during the Devonian Period of the Paleozoic Era.']\",\"question\":\"When did the first mammals appear? Compare the monotremesand marsupials.\",\"src\":\"stemez-Biology\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}