{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"9b6e784e-3c3a-5fdc-a783-8ad3cf1a5a67","task_key":"default--test--7488","task_revision_id":"1","upstream_id":"7488","short_description":"Which of the following statements is correct in regard to the federal budget…","config":"default","split":"test","body":"{\"category\":\"economics\",\"options\":\"['The debt is the sum of all past and future deficits.', 'When the debt is positive the deficit decreases.', 'When the deficit is negative the debt decreases.', 'The deficit is the accumulation of past debts.', 'When the debt is negative the deficit decreases.', 'The deficit is the accumulation of future debts.', 'The deficit and the debt are always equal.', 'When the deficit is positive the debt decreases.', 'When the debt is positive the deficit increases.', 'When the debt is negative the deficit increases.']\",\"question\":\"Which of the following statements is correct in regard to the federal budget deficit and the federal debt?\",\"src\":\"ori_mmlu-high_school_macroeconomics\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}