{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"848e5994-c314-57f3-a393-0761db4007e9","task_key":"default--test--10749","task_revision_id":"1","upstream_id":"10749","short_description":"A large data set contains information about all students majoring in computer…","config":"default","split":"test","body":"{\"category\":\"computer science\",\"options\":\"['What is the percentage of students majoring in computer science who also participate in sports?', 'How many states have a higher percentage of female computer science majors than male computer science majors attending college in that state?', 'Do students majoring in computer science tend to have higher grade point averages than students majoring in other subjects?', 'What is the average age of students majoring in computer science?', 'What is the most common second major for students majoring in computer science?', 'Which college has the highest number of students majoring in computer science?', 'What percent of computer science majors receive financial aid?', 'What percent of students attending college in a certain state are majoring in computer science?']\",\"question\":\"A large data set contains information about all students majoring in computer science in colleges across the United States. The data set contains the following information about each student.\\n   ° The student's gender\\n   ° The state in which the student attends college\\n   ° The student's grade point average on a 4.0 scale\\n\\n Which of the following questions could be answered by analyzing only information in the data set?\",\"src\":\"ori_mmlu-high_school_computer_science\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}