{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"564f31de-7b98-5e67-841d-b75704450a8a","task_key":"default--test--5080","task_revision_id":"1","upstream_id":"5080","short_description":"The case of the former Soviet Union supports Kerr et al.'s (1960) claim that:","config":"default","split":"test","body":"{\"category\":\"other\",\"options\":\"['advanced industrial technology strengthens the power of totalitarian regimes', 'communist societies are more likely to disperse power between various economic classes', 'democratic governments hinder the development of advanced industrial technology', 'advanced industrial technology has no impact on the structure of totalitarian regimes', 'totalitarian regimes discourage the development of a ruling elite', 'communist societies are more likely to adopt advanced industrial technology', 'totalitarian regimes are more likely to maintain power with the help of advanced industrial technology', 'democratic governments serve the interests of a ruling elite', 'totalitarian regimes are incompatible with the requirements of advanced industrial technology', 'communist societies disperse power between various political parties']\",\"question\":\"The case of the former Soviet Union supports Kerr et al.'s (1960) claim that:\",\"src\":\"ori_mmlu-sociology\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}