{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"822a4a23-2601-5878-8aaa-b4d566bea725","task_key":"default--test--8262","task_revision_id":"1","upstream_id":"8262","short_description":"Information about what the students in Mr. Paulson’s class and in Ms. Hugo’s…","config":"default","split":"test","body":"{\"category\":\"math\",\"options\":\"['the total number of pizza slices and hamburgers eaten that day', 'the grades of the students in both classes', 'the dietary preferences of the students in both classes', 'the types of pizza and hamburgers eaten', 'the total number of students in each teacher’s class', 'nothing because the portions each class ate were the same', 'whether the lunches were eaten on the same day', 'the number of students who did not eat lunch in each class', 'the time at which lunch was eaten in each class', 'the gender ratio in each class']\",\"question\":\"Information about what the students in Mr. Paulson’s class and in Ms. Hugo’s class ate for lunch is listed below. In Mr. Paulson’s class, 0.5 of the students ate pizza for lunch. In Ms. Hugo’s class, 0.5 of the students ate hamburgers for lunch. What information, if any, is needed to correctly compare the 0.5 of Mr. Paulson’s class that ate pizza to the 0.5 of Ms. Hugo’s class that ate hamburgers?\",\"src\":\"ori_mmlu-elementary_mathematics\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}