{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"181b6b53-3b7a-53d5-b8a2-5fd3b6a6626c","task_key":"default--test--7475","task_revision_id":"1","upstream_id":"7475","short_description":"Adam Smith described in great detail the superior produc-tivity of the division…","config":"default","split":"test","body":"{\"category\":\"economics\",\"options\":\"['The overall population size', 'the level of technology', 'The geographic distribution of resources', 'The number of different jobs in the economy', 'the size of the prospective markets', 'the availability of capital', 'the skill level of the labor force', \\\"The education system's effectiveness\\\", 'The cultural attitudes towards work', 'Government regulations and policies']\",\"question\":\"Adam Smith described in great detail the superior produc-tivity of the division of labor. What did he thinkdivision of labor was limited by?\",\"src\":\"stemez-Economics\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}