{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"504109af-3b4f-5e7d-b854-9214aaaaf911","task_key":"default--test--7052","task_revision_id":"1","upstream_id":"7052","short_description":"Suppose that aluminum is a key production input in the production of bicycles.…","config":"default","split":"test","body":"{\"category\":\"economics\",\"options\":\"['the supply of aluminum to fall.', 'the price of bicycles to fall.', 'the supply of bicycles to fall.', 'the supply of bicycles to rise.', 'the demand for bicycles to rise.', 'the demand for bicycles to fall.', 'the demand for aluminum to fall.', 'the supply of aluminum to rise.', 'the demand for aluminum to rise.', 'the price of bicycles to rise.']\",\"question\":\"Suppose that aluminum is a key production input in the production of bicycles. If the price of aluminum falls, and all other variables are held constant, we expect\",\"src\":\"ori_mmlu-high_school_microeconomics\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}