{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"c689eef0-851d-53b7-8bb6-e451013faf24","task_key":"default--test--1154","task_revision_id":"1","upstream_id":"1154","short_description":"A police officer entered a private home in response to a reported disturbance.…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"[\\\"The federal constitution cannot be used to interfere with a state's right to regulate sexual morality.\\\", \\\"The statute is unconstitutional because it infringes on the men's Ninth Amendment rights, which protect other rights not specifically enumerated in the Constitution.\\\", \\\"The statute is unconstitutional because it infringes the men's right to equal protection under the Fourteenth Amendment.\\\", \\\"The statute is unconstitutional primarily because it infringes the men's right to freedom of expression under the First Amendment.\\\", \\\"The relationship in question is not protected by the right to privacy and is subject to a state's criminal regulation.\\\", 'The statute is constitutional as it promotes the common good.', \\\"The statute violates the men's right to liberty encompassed within the constitutional guarantee of substantive due process.\\\", \\\"The statute is constitutional as it is based on the state's interpretation of religious principles.\\\", \\\"The statute is unconstitutional as it goes against the Eighth Amendment's prohibition of cruel and unusual punishment.\\\", \\\"The statute is constitutional because it falls within the state's police power to regulate health, safety, welfare, and morals.\\\"]\",\"question\":\"A police officer entered a private home in response to a reported disturbance. She found no disturbance but did witness the two male residents engaging in consensual sexual relations with each other. She arrested and charged them with \\\"deviate sexual intercourse\\\", a state law that prohibits two persons of the same sex from engaging in mutual sexual relations. After conviction and unsuccessful appeals in the state courts, the men appealed to the United States Supreme Court. What is the likely holding and reasoning of the Supreme Court?\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}