{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"15056311-7c55-521d-b199-c65b5138c975","task_key":"default--test--7041","task_revision_id":"1","upstream_id":"7041","short_description":"The apple market is in equilibrium. Suppose we observe that apple growers are…","config":"default","split":"test","body":"{\"category\":\"economics\",\"options\":\"['Price is ambiguous, but quantity rises.', 'Price and quantity remain the same.', 'Quantity rises, but price falls.', 'Price and quantity both rise.', 'Price and quantity both fall.', 'Price rises, but quantity is ambiguous.', 'Price rises, quantity falls.', 'Price is ambiguous, but quantity falls.', 'Price falls, but quantity is ambiguous.', 'Price falls, quantity rises.']\",\"question\":\"The apple market is in equilibrium. Suppose we observe that apple growers are using more pesticides to increase apple production. At the same time, we hear that the price of pears, a substitute for apples, is rising. Which of the following is a reasonable prediction for the new price and quantity of apples?\",\"src\":\"ori_mmlu-high_school_microeconomics\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}