{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"a3e4095f-7860-530f-bbda-c02d85d0e061","task_key":"default--test--8341","task_revision_id":"1","upstream_id":"8341","short_description":"In the jury pool available for this week, 30 percent of potential jurors are…","config":"default","split":"test","body":"{\"category\":\"math\",\"options\":\"['0.700', 'The answer cannot be determined without knowing the size of the jury pool.', '0.168', '0.550', '0.915', 'The answer can be determined without knowing the size of the jury pool.', '0.300', '0.457', 'The answer cannot be determined without knowing the percentage of men in the jury pool.', '0.843']\",\"question\":\"In the jury pool available for this week, 30 percent of potential jurors are women. A particular trial requires that, out of a jury of 12, at least three are women. If a jury of 12 is to be selected at random from the pool, what is the probability it meets the requirements of this trial?\",\"src\":\"ori_mmlu-high_school_statistics\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}