{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"b154e1d9-1c0d-5d8e-9d49-ad93a02f24b7","task_key":"default--test--7164","task_revision_id":"1","upstream_id":"7164","short_description":"An increase in corporate optimism will have which of the following effects in…","config":"default","split":"test","body":"{\"category\":\"economics\",\"options\":\"['A decrease in both supply and demand and an ambiguous change in interest rates.', 'An increase in supply lowering the interest rate.', 'A decrease in demand increasing the interest rate.', 'An increase in demand increasing the interest rate.', 'A decrease in supply increasing the interest rate.', 'An increase in supply and a decrease in demand, lowering the interest rate.', 'No significant change in the market for loanable funds.', 'A decrease in both supply and demand, lowering the interest rate.', 'An increase in both supply and demand and an ambiguous change in interest rates.', 'A decrease in demand lowering the interest rate.']\",\"question\":\"An increase in corporate optimism will have which of the following effects in the market for loanable funds?\",\"src\":\"ori_mmlu-high_school_macroeconomics\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}