{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"0f69839d-f7b4-5f0a-b4f1-01625764193c","task_key":"default--test--5813","task_revision_id":"1","upstream_id":"5813","short_description":"A triad on a color wheel can be described as","config":"default","split":"test","body":"{\"category\":\"other\",\"options\":\"['three colors with the same hue', 'a pair of complementary colors', 'three colors opposite each other on the color wheel', 'three colors next to each other on the color wheel', 'three colors forming a pentagon on the color wheel', 'three colors of equal value', 'three colors equally spaced apart', 'a complementary set', 'three analogous colors', 'three colors forming a square on the color wheel']\",\"question\":\"A triad on a color wheel can be described as\",\"src\":\"ori_mmlu-miscellaneous\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}