{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"41e88236-bb7e-5520-a0c6-5e6d34da6a5d","task_key":"all--test--41e88236-bb7e-5520-a0c6-5e6d34da6a5d","task_revision_id":"1","upstream_id":"","short_description":"A professor employed by a state university is a well- known critic of foreign…","config":"all","split":"test","body":"{\"choices\":\"['not prevail, because the criticism was not directed at the professor personally.', 'not prevail, because the broadcast was privileged as being in the public interest.', 'prevail, because the professor, as a private individual, was placed in a false light.', 'prevail, because the comments were made with actual malice.']\",\"question\":\"A professor employed by a state university is a well- known critic of foreign policy and has sometimes publicly stated that he wished for the United States to suffer some collective political tragedy in retribution for all the harms it does to the world. The professor was recently involved in a highly visible political protest against the government. The police eventually shut down the protest because they felt that it was becoming unruly. A television crew covered the entire rally. On the six o'clock news that evening, the film of the protest rally was shown during the broadcast. A news anchorman then made the following commentary: \\\"It's a shame that public funds are spent to pay the salaries of such university professors, who are not grateful for what this country has done for them. In my opinion, these people like the professor should be deported. \\\"If the professor asserts a claim based on invasion of privacy against the television station and the anchorman for his television commentary, the most likely result is that the professor will\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}