{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"9652dc07-9644-5805-a245-2f5992edf4a7","task_key":"all--test--9652dc07-9644-5805-a245-2f5992edf4a7","task_revision_id":"1","upstream_id":"","short_description":"On the basis of scientific studies showing a causal relationship between the…","config":"all","split":"test","body":"{\"choices\":\"['No, because it does not serve a substantial government interest.', 'No, because it is more extensive than necessary to serve the government interest in preventing certain cancers.', 'Yes, because it does not affect speech protected by the First Amendment.', 'Yes, because it serves a legitimate government interest in protecting public health.']\",\"question\":\"On the basis of scientific studies showing a causal relationship between the consumption of \\\"red meat\\\" (principally beef) and certain forms of cancer, a federal statute prohibits all commercial advertising of red meat products. The statute does not, however, restrict the sale of red meat products. Producers of red meat have challenged the statute as a violation of their free speech rights protected by the First Amendment. Is the court likely to find the statute constitutional?\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}