{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"e3761d5b-dd6d-5d5f-a36c-f552f3571e3c","task_key":"default--test--6762","task_revision_id":"1","upstream_id":"6762","short_description":"A new severe respiratory illness caused by a newly identified virus is…","config":"default","split":"test","body":"{\"category\":\"health\",\"options\":\"['Is more likely to provide sterilizing immunity', 'Can be administered in higher doses', 'Develops more rapid protective immunity', 'Is less likely to require subsequent boosters for lifelong immunity', 'Is less likely to cause an allergic reaction', 'Is more effective for immunocompromised individuals', 'Is most likely to generate mucosal immunity', 'Is less likely to cause local adverse reactions at the site of injection', 'Avoids the concern for reversion to virulence', 'Is less likely to be affected by preexisting antibodies']\",\"question\":\"A new severe respiratory illness caused by a newly identified virus is discovered. Which of the following properties of a killed vaccine relative to a live vaccine is the most appropriate rationale for developing a killed vaccine for this illness?\",\"src\":\"ori_mmlu-professional_medicine\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}