{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"32820ae2-7ec6-5ea4-a91c-975274060a9c","task_key":"default--test--7757","task_revision_id":"1","upstream_id":"7757","short_description":"Suppose a fair coin is tossed 50 times. The bound on the probability that the…","config":"default","split":"test","body":"{\"category\":\"math\",\"options\":\"['0.075', '0.250', '0.150', '0.175', '0.100', '0.225', '0.125', '0.050', '0.300', '0.200']\",\"question\":\"Suppose a fair coin is tossed 50 times. The bound on the probability that the number of heads will be greater than 35 or less than 15 can be found using Chebyshev's Inequality. What is the upper bound of the probability?\",\"src\":\"theoremQA-Math\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}