{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"f6775a03-78f9-5ec2-97e3-a8615b933cd7","task_key":"default--test--1506","task_revision_id":"1","upstream_id":"1506","short_description":"A woman belonged to an organization that advocated for the United States to…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['The woman held a leadership position in the organization.', 'The woman expressed public support of the organization after the violent demonstrations.', 'The woman recruited new members into the organization after the violent demonstrations.', 'The woman was physically present during the violent demonstrations.', 'The woman continued to provide material aid to the organization through the payment of dues after the violent demonstrations.', 'The woman provided material aid to the organization through the payment of dues before the violent demonstrations.', \\\"The woman had knowledge of the organization's violent activities but did not report them.\\\", 'The woman joined the organization with the specific intent of furthering its illegal activities.', \\\"The woman had knowledge of the organization's illegal activities before joining.\\\", \\\"The woman publicly defended the organization's violent activities.\\\"]\",\"question\":\"A woman belonged to an organization that advocated for the United States to preemptively attack certain foreign countries. The organization usually used leafleting and public speeches to advance this view, but it sometimes engaged in violent demonstrations against the embassies and consuls of those countries. Although the woman had never participated in a violent demonstration, she was criminally prosecuted for being a member of the organization. In her defense, the woman claimed that her association with the organization was protected by the First Amendment. Which of the following would the prosecution need to prove to overcome that defense?\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}