{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"31407bb5-9221-57c7-9acd-ee3e77c6a73b","task_key":"default--test--7509","task_revision_id":"1","upstream_id":"7509","short_description":"The United States is trading salmon to Peru in exchange for anchovies. If these…","config":"default","split":"test","body":"{\"category\":\"economics\",\"options\":\"['Both the United States and Peru have comparative advantage in salmon production.', 'The United States has absolute advantage in both salmon and anchovy production.', 'Both the United States and Peru have comparative advantage in anchovy production.', 'The United States has comparative advantage in both salmon and anchovy production.', 'The United States has comparative advantage in salmon production, and Peru has comparative advantage in anchovy production.', 'Peru has comparative advantage in both salmon and anchovy production.', 'The United States has absolute advantage in anchovy production, and Peru has absolute advantage in salmon production.', 'Peru has absolute advantage in both salmon and anchovy production.', 'The United States has absolute advantage in salmon production, and Peru has absolute advantage in anchovy production.', 'The United States has comparative advantage in anchovy production, and Peru has comparative advantage in salmon production.']\",\"question\":\"The United States is trading salmon to Peru in exchange for anchovies. If these nations are trading based upon relative opportunity costs, what must be the case?\",\"src\":\"ori_mmlu-high_school_microeconomics\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}