{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"9a57cc4c-b21a-59ac-ac07-b3a6a5ad0977","task_key":"default--test--1647","task_revision_id":"1","upstream_id":"1647","short_description":"Liang, a talented student from another country, received degrees from several…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['No, there is no compelling state interest.', 'Yes, because the teacher could exert too much influence.', 'No, because the Equal Protection Clause would protect him.', 'Yes, because states can regulate aliens.', 'Yes, because the state has the right to set its own educational policies.', 'Yes, because the state has an interest in ensuring that teachers understand and can teach American values.', 'No, because the state cannot discriminate against non-citizens.', 'No, because a state cannot discriminate based on nationality.', 'No, unless they use the rationally related to a legitimate state interest standard.', 'Yes, because the state can set its own employment criteria.']\",\"question\":\"Liang, a talented student from another country, received degrees from several universities in the United States and then decided to stay in the states to teach. Could a state law requiring citizenship for teachers prevent him from doing so?\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}