{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"cc75ebec-4c54-548d-a506-42c6ef822362","task_key":"default--test--3153","task_revision_id":"1","upstream_id":"3153","short_description":"A microbiologist takes a stool specimen from you and a few days later tells you…","config":"default","split":"test","body":"{\"category\":\"biology\",\"options\":\"['Only pathogenic strains of both Escherichia coli and Salmonella typhi', 'Neither Escherichia coli nor Salmonella typhi', 'Only nonpathogenic strains of Salmonella typhi', 'Only pathogenic strains of Escherichia coli', 'Both Escherichia coli and Salmonella typhi', 'Salmonella typhi', 'Escherichia coli O157:H7', 'Only nonpathogenic strains of both Escherichia coli and Salmonella typhi', 'Escherichia coli', 'Any Escherichia coli not part of the normal gut flora']\",\"question\":\"A microbiologist takes a stool specimen from you and a few days later tells you that Escherichia coli andSalmonella typhi are growing in your intestinal tract. Which typeshould you be concerned about?\",\"src\":\"stemez-Biology\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}