{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"f4935347-f43d-56d1-9d35-2d31902cdef8","task_key":"default--test--6683","task_revision_id":"1","upstream_id":"6683","short_description":"A 22-year-old male presents to the office with a 5-day history of diarrhea after…","config":"default","split":"test","body":"{\"category\":\"health\",\"options\":\"['Shigella dysenteriae', 'Campylobacter jejuni', 'Giardia lamblia', 'Staphylococcus aureus', 'Entamoeba histolytica', 'Yersinia enterocolitica', 'Clostridium difficile', 'Escherichia coli', 'Vibrio cholerae', 'Salmonella typhi']\",\"question\":\"A 22-year-old male presents to the office with a 5-day history of diarrhea after completing his third course of antibiotics for mastoiditis. Physical examination reveals vague generalized abdominal pain on palpation. Culture on hektoen enteric agar is positive. The most likely etiologic agent causing the diarrhea is \",\"src\":\"ori_mmlu-professional_medicine\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}