{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"8f6ac460-e1d6-508a-8ac2-9f93968baf54","task_key":"default--test--7572","task_revision_id":"1","upstream_id":"7572","short_description":"Suppose the market for roses is currently in equilibrium. If the supply of roses…","config":"default","split":"test","body":"{\"category\":\"economics\",\"options\":\"['Both price and quantity change is ambiguous.', 'Price falls and quantity rises.', 'Price and quantity both rise.', 'Price and quantity both fall.', 'Price rises, but the change in quantity is ambiguous.', 'Price and quantity remain the same.', 'Quantity falls, but the change in price is ambiguous.', 'Quantity rises, but the change in price is ambiguous.', 'Price rises and quantity falls.', 'Price falls, but the change in quantity is ambiguous.']\",\"question\":\"Suppose the market for roses is currently in equilibrium. If the supply of roses falls, while at the same time the demand for roses rises, what can you say about the price and quantity of roses in the market?\",\"src\":\"ori_mmlu-high_school_microeconomics\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}