{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"9e689a2d-d739-56cb-90a5-a10f9958595f","task_key":"default--test--6912","task_revision_id":"1","upstream_id":"6912","short_description":"Which one of the following factors is likely to lead to a relatively high degree…","config":"default","split":"test","body":"{\"category\":\"economics\",\"options\":\"['A model that contains many variables', 'A model that is based on financial theory', 'A model whose dependent variable has recently exhibited a structural change', 'A model that heavily relies on subjective judgment', 'A model that does not consider economic indicators', 'A model that is entirely statistical in nature with no room for judgmental modification of forecasts', 'A model that is based on a single variable', 'A model that predicts based on historical trends only', 'A model with a high degree of complexity', 'A model that ignores market volatility']\",\"question\":\"Which one of the following factors is likely to lead to a relatively high degree of out-of-sample forecast accuracy?\",\"src\":\"ori_mmlu-econometrics\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}