{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"b75c4889-0b67-5804-a6b0-39e2a6686387","task_key":"default--test--8284","task_revision_id":"1","upstream_id":"8284","short_description":"Mr. Cleary’s class and Ms. Ntuala’s class go to use the computer lab. There are…","config":"default","split":"test","body":"{\"category\":\"math\",\"options\":\"['2', '18', '8', '6', '16', '12', '20', '10', '24', '14']\",\"question\":\"Mr. Cleary’s class and Ms. Ntuala’s class go to use the computer lab. There are 20 computers available, two of which do not work. Mr. Cleary’s class has 14 kids, and Ms. Ntuala’s class has 12 kids. If every student must use a computer and there can only be 2 students on a computer at most, what is the maximum number of students who can have a computer to themselves?\",\"src\":\"ori_mmlu-high_school_mathematics\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}