{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"7c0498e7-460e-55d5-9fe9-829ab83b35d8","task_key":"default--test--6763","task_revision_id":"1","upstream_id":"6763","short_description":"A new severe respiratory illness caused by a newly identified virus is…","config":"default","split":"test","body":"{\"category\":\"health\",\"options\":\"['Is less likely to cause local adverse reactions at the site of injection', 'Is most likely to generate mucosal immunity', 'Is less likely to be affected by preexisting antibodies', 'Can be administered in higher doses', 'Develops more rapid protective immunity', 'Is less likely to cause an allergic reaction', 'Avoids the concern for reversion to virulence', 'Is less likely to require subsequent boosters for lifelong immunity', 'Is more likely to provide sterilizing immunity', 'Is more effective for immunocompromised individuals']\",\"question\":\"A new severe respiratory illness caused by a newly identified virus is discovered. Which of the following properties of a killed vaccine relative to a live vaccine is the most appropriate rationale for developing a killed vaccine for this illness? \",\"src\":\"ori_mmlu-professional_medicine\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}