{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"9cafa567-935c-5f36-9edc-31995ace1c77","task_key":"default--test--6841","task_revision_id":"1","upstream_id":"6841","short_description":"A student decides that, having already spent three hours studying for an exam,…","config":"default","split":"test","body":"{\"category\":\"economics\",\"options\":\"['The marginal benefit of the third hour was likely greater than the marginal benefit of the fourth hour.', 'The marginal cost of the third hour was likely greater than the marginal cost of the fourth hour.', 'The marginal benefit of the fourth hour is at least as great as the marginal cost of the fourth hour.', 'The marginal benefit of the fourth hour is certainly less than the marginal cost of the fourth hour.', 'The marginal cost of the fourth hour is at least as great as the marginal benefit of the fourth hour.', 'The marginal cost of the fourth hour is likely less than the marginal cost of the third hour.', 'The marginal benefit of the fourth hour is certainly more than the marginal cost of the fourth hour.', 'Without knowing the student’s opportunity cost of studying, we have no way of knowing whether or not her marginal benefits outweigh her marginal costs.', \\\"Without knowing the student's other commitments, we cannot determine the marginal cost or benefit of the fourth hour.\\\"]\",\"question\":\"A student decides that, having already spent three hours studying for an exam, she should spend one more hour studying for the same exam. Which of the following is most likely true?\",\"src\":\"ori_mmlu-high_school_microeconomics\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}