{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"4c22206d-c217-539d-8188-0b6c4dfda0c7","task_key":"default--test--1689","task_revision_id":"1","upstream_id":"1689","short_description":"A professor employed by a state university is a well- known critic of foreign…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['not prevail, because the broadcast was privileged as being in the public interest.', \\\"prevail, because the comments were intrusive and violated the professor's privacy.\\\", 'not prevail, because the criticism was not directed at the professor personally.', \\\"not prevail, because the professor's public behavior and statements made him a public figure.\\\", 'prevail, because the professor, as a private individual, was placed in a false light.', \\\"not prevail, because the comments were based on the professor's public actions and statements.\\\", \\\"prevail, because the comments were defamatory and damaged the professor's reputation.\\\", 'prevail, because the comments were made with actual malice.', 'not prevail, because the professor willingly participated in the public protest.', \\\"prevail, because the anchorman's comments were excessive and unnecessary.\\\"]\",\"question\":\"A professor employed by a state university is a well- known critic of foreign policy and has sometimes publicly stated that he wished for the United States to suffer some collective political tragedy in retribution for all the harms it does to the world. The professor was recently involved in a highly visible political protest against the government. The police eventually shut down the protest because they felt that it was becoming unruly. A television crew covered the entire rally. On the six o'clock news that evening, the film of the protest rally was shown during the broadcast. A news anchorman then made the following commentary: \\\"It's a shame that public funds are spent to pay the salaries of such university professors, who are not grateful for what this country has done for them. In my opinion, these people like the professor should be deported. \\\"If the professor asserts a claim based on invasion of privacy against the television station and the anchorman for his television commentary, the most likely result is that the professor will \",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}