{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"468ce9ca-e446-5bde-91b1-2e388d397620","task_key":"default--test--1185","task_revision_id":"1","upstream_id":"1185","short_description":"A state in which several popular ski resorts are located has enacted a law…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['Yes, because the requirement substantially impairs her ability to contract for employment in the absence of an important government objective.', 'Yes, because the law was enacted after she moved to the state.', 'No, because the law is necessary for the safety of skiers.', 'No, because her ability to contract for employment with the state is not being impaired.', 'Yes, because the law restricts her freedom to engage in her profession.', 'Yes, because, as applied to an otherwise qualified ski instructor, the requirement is not rationally related to a legitimate government interest.', 'Yes, because the certification requirement is too costly and time-consuming.', 'No, because the certification requirement is intended to standardize ski instruction.', 'No, because she has no existing contract that is being impaired.', 'No, because the law applies to all ski instructors equally.']\",\"question\":\"A state in which several popular ski resorts are located has enacted a law requiring state certification of ski instructors. Under the law, applicants for certification must attend a monthlong course, pass a test, and pay a fee of $1,000. The stated purpose of the law is to \\\"promote uniformity in the methods of ski instruction at the state's ski areas.\\\" Shortly before the law's enactment, when the state did not require certification of ski instructors, a woman moved to the state intending to find employment as a ski instructor. The woman had been a ski instructor in another state for many years. But since her move, even though ski resorts in the state have found her to be highly qualified, no resort will offer her a contract for employment as a ski instructor because she has not received the required state certification. As applied to the woman, does the state certification requirement constitute a violation of the contracts clause?\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}