{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"a6cee617-b022-5235-9736-7c3aa8c2f329","task_key":"default--test--3121","task_revision_id":"1","upstream_id":"3121","short_description":"How do cyanobacteria differ from green unicellular algae?","config":"default","split":"test","body":"{\"category\":\"biology\",\"options\":\"['Cyanobacteria reduce sulfur compounds.', 'Cyanobacteria have no nuclei.', 'Green algae cannot reduce nitrogen compounds.', 'Green algae cannot photosynthesize.', 'Green algae produce cell membranes.', 'Green algae have no chlorophyll.', 'Cyanobacteria lack cell walls.', 'Cyanobacteria can photosynthesize.']\",\"question\":\"How do cyanobacteria differ from green unicellular algae?\",\"src\":\"ori_mmlu-college_biology\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}