{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"63a0fc5b-f0b3-5349-b2f1-4e20ca14dadc","task_key":"default--test--2539","task_revision_id":"1","upstream_id":"2539","short_description":"Maria, a bright high school student, fears success. To which of the following…","config":"default","split":"test","body":"{\"category\":\"psychology\",\"options\":\"['Harvard, the most competitive state college in her state, and distant community colleges', 'The most competitive state college in her state and local community college', 'Stanford, Oxford, and the most competitive state college in her state', 'Harvard, Yale, and Stanford', 'the local community college and distant community colleges', 'The least competitive state college in her state and distant community colleges', 'Oxford, Cambridge, and the least competitive state college in her state', 'Harvard, Stanford, and the local community college', 'Harvard, Stanford, and the least competitive state college in her state', 'Stanford, Oxford, and the least competitive state college in her state']\",\"question\":\"Maria, a bright high school student, fears success. To which of the following colleges would she most likely apply?\",\"src\":\"ori_mmlu-high_school_psychology\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}