{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"b20d3dae-a89a-5568-8b93-b269b000ddc4","task_key":"default--test--6903","task_revision_id":"1","upstream_id":"6903","short_description":"At the birthday party of your best friend, you see Skylar help himself to a…","config":"default","split":"test","body":"{\"category\":\"economics\",\"options\":\"['the marginal benefit of the second piece of cake is less than the marginal cost.', 'the total benefit received from eating cake is falling.', 'the ratio of marginal benefit over marginal cost is less than one.', 'the marginal benefit of the second piece of cake is greater than the marginal cost.']\",\"question\":\"At the birthday party of your best friend, you see Skylar help himself to a second piece of cake. For this individual, it must be the case that\",\"src\":\"ori_mmlu-high_school_microeconomics\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}