{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"586fb0be-609c-5ac1-81cc-12624df8821b","task_key":"default--test--6836","task_revision_id":"1","upstream_id":"6836","short_description":"When the price of pears increases, we expect the following:","config":"default","split":"test","body":"{\"category\":\"economics\",\"options\":\"['Quantity supplied of pears falls.', 'The price of apples falls.', 'There is no change in the quantity demanded or supplied.', 'The price of pears falls.', 'Quantity demanded of pears falls.', 'There is a surplus of pears in the market.', 'Quantity demanded of pears rises.', 'Demand for pears falls.', 'Quantity supplied of pears rises.', 'Demand for pears rises.']\",\"question\":\"When the price of pears increases, we expect the following:\",\"src\":\"ori_mmlu-high_school_microeconomics\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}