{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"c1fe4e8e-d670-5c16-bb3a-1d1b798f659e","task_key":"default--test--10938","task_revision_id":"1","upstream_id":"10938","short_description":"Rawls argues that parties in the original position would not accept…","config":"default","split":"test","body":"{\"category\":\"philosophy\",\"options\":\"['it is too difficult to determine which policies would maximize happiness.', 'doing so would be too risky.', 'it requires us to do unjust things, such as executing innocent people.', 'they would be behind the veil of ignorance and thus would never have heard of utilitarianism.']\",\"question\":\"Rawls argues that parties in the original position would not accept utilitarianism because:\",\"src\":\"ori_mmlu-philosophy\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}