{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"48a7a486-7f35-5588-abe6-85a18b50e534","task_key":"default--test--5873","task_revision_id":"1","upstream_id":"5873","short_description":"Pluralist theories suggest that:","config":"default","split":"test","body":"{\"category\":\"other\",\"options\":\"[\\\"political parties must compete for the votes of 'consumers' in the electorate\\\", 'political parties are primarily focused on maintaining power rather than serving the electorate', \\\"the state's power is primarily exercised through a centralized administrative structure\\\", 'there is a direct correlation between wealth and political power', 'the ruling elite is composed of people from various class backgrounds', 'the ruling elite is composed solely of individuals from high class backgrounds', 'power is concentrated in the hands of a few elite individuals or groups', 'there is a close alignment between class background and party preference', \\\"the state's power can be exercised through several different administrative structures\\\"]\",\"question\":\"Pluralist theories suggest that:\",\"src\":\"ori_mmlu-sociology\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}