{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"4d590075-163f-52d7-b4a0-9408d0423924","task_key":"default--test--5328","task_revision_id":"1","upstream_id":"5328","short_description":"A student in the process of solving a fabrication problem in the manufacturing…","config":"default","split":"test","body":"{\"category\":\"other\",\"options\":\"['tell the student to figure it out independently without any suggestions', 'instruct the student to follow the same procedure used in the previous project', 'place the responsibility completely on the student for making the judgment', 'recommend the student to read the textbook or watch online tutorials', 'provide a step-by-step guide for the student to follow without explanation', 'use leading questions to help the student review and analyze the relative merits of several assembly procedures', 'have another student solve the problem while the asking student observes', 'give an opinion as to the best assembly procedure for the particular problem', 'suggest two or three possible assembly procedures and have the student select one', 'ignore the question and move on to the next task']\",\"question\":\"A student in the process of solving a fabrication problem in the manufacturing laboratory asks the teacher what assembly procedures should be used. The teacher's best response would be to\",\"src\":\"ori_mmlu-miscellaneous\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}