{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"a752e114-2f3e-5027-af4f-816ad7a23495","task_key":"default--test--7797","task_revision_id":"1","upstream_id":"7797","short_description":"There are two games involving flipping a fair coin. In the first game you win a…","config":"default","split":"test","body":"{\"category\":\"math\",\"options\":\"['200 times for the first game and 100 times for the second', '300 times for each game', '30 times for the first game and 300 times for the second', '300 times for the first game and 30 times for the second', '250 times for the first game and 50 times for the second', '100 times for the first game and 200 times for the second', '30 times for each game', '100 times for each game', '50 times for the first game and 250 times for the second', '150 times for each game']\",\"question\":\"There are two games involving flipping a fair coin. In the first game you win a prize if you can throw between 45% and 55% heads. In the second game you win if you can throw more than 80% heads. For each game would you rather flip the coin 30 times or 300 times?\",\"src\":\"ori_mmlu-high_school_statistics\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}