{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"2a2d8bd6-cac4-5001-bbc6-84be7a660fe2","task_key":"default--test--7439","task_revision_id":"1","upstream_id":"7439","short_description":"Which of the following is true of the perfectly competitive firm in the short…","config":"default","split":"test","body":"{\"category\":\"economics\",\"options\":\"['The firm always operates at a loss in the short run.', \\\"The firm's profits are not affected by average total cost in the short run.\\\", 'The firm earns a normal profit.', 'The firm shuts down if the price falls below average total cost.', 'The firm will always shut down if there is any drop in price.', 'The firm earns positive economic profit.', 'The firm may earn positive, negative, or normal profits.', 'The firm will always earn a negative economic profit in the short run.', 'The firm cannot earn a positive economic profit in the short run.', \\\"The firm's profits do not depend on market conditions.\\\"]\",\"question\":\"Which of the following is true of the perfectly competitive firm in the short run?\",\"src\":\"ori_mmlu-high_school_microeconomics\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}