{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"62455b1d-033a-529f-8d6c-3de3bd858b16","task_key":"default--test--1488","task_revision_id":"1","upstream_id":"1488","short_description":"Which statement is closest to characterizing the principal difference between…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['They differ in their views about the necessity of a social contract.', 'They disagree about the role of law in society.', 'They have contrasting views on the role of religion in society.', 'They have different ideas about the importance of education.', 'They have different perspectives on economic policies.', 'They have opposing views about the nature of contractual obligations.', 'They differ in respect of their account of life before the social contract.', 'They hold opposing views on the subject of individual rights.', 'They disagree on the structure and role of government in society.', 'They adopt different attitudes towards the role of the courts in maintaining order.']\",\"question\":\"Which statement is closest to characterizing the principal difference between the positions adopted by Hobbes and Locke?\",\"src\":\"ori_mmlu-jurisprudence\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}