{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"66159ebf-27bb-5979-ad9a-6ce1742d27a8","task_key":"default--test--1032","task_revision_id":"1","upstream_id":"1032","short_description":"A state legislature enacts a statute prohibiting the sale of contraceptive…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['not to have standing.', 'to have standing.', 'to have standing, but only if the patients also sue', 'not to have standing jus tertii.', 'not to have standing, unless the patients join the lawsuit', 'not to have standing, unless he can prove the health risk', 'not to have standing, because the law does not directly affect him', 'to have standing jus tertii.', 'to have standing, but only if he can prove the health risk', 'to have standing, but not on behalf of his patients']\",\"question\":\"A state legislature enacts a statute prohibiting the sale of contraceptive devices to married persons. This state statute prohibits the use of contraceptive devices by married persons. A physician who practices in the state brings suit in federal court challenging the constitutionality of the state contraceptive statute. The physician attacks the validity of the statute on the grounds that it prevents him from giving professional advice concerning the use of contraceptives to three patients, all of whom are married, whose condition of health might be endangered by child bearing. The plaintiff is likely\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}