{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"09a25e29-ccde-5203-8ce5-3721286c3899","task_key":"default--test--11228","task_revision_id":"1","upstream_id":"11228","short_description":"Dershowitz discusses the hypothetical case where the hijacked planes of 9/11…","config":"default","split":"test","body":"{\"category\":\"philosophy\",\"options\":\"['should always be made by a single person to ensure accountability.', 'are often influenced by personal bias.', 'should be made by the military without civilian interference.', 'should be made by a committee of experts.', 'can be avoided through diplomatic negotiations.', 'have no objective answer.', 'none of the above', 'are not worth the moral cost.', 'often have to be made quickly and without immediate authorization.', 'should be made with visibility and accountability by the highest-level officials.']\",\"question\":\" Dershowitz discusses the hypothetical case where the hijacked planes of 9/11 must be shot down to argue that important decisions like these\",\"src\":\"ori_mmlu-moral_disputes\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}