{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"5584c664-a75f-56a0-8d41-1e5def88e66d","task_key":"default--test--11153","task_revision_id":"1","upstream_id":"11153","short_description":"Nagel claims that prior to reflection, it is plausible that people cannot be…","config":"default","split":"test","body":"{\"category\":\"philosophy\",\"options\":\"['actions that affect only themselves.', 'actions performed under duress.', 'actions that do not violate anyone’s rights.', 'actions that benefit others.', 'what is due to factors beyond their control.', 'actions that are socially acceptable.', 'their personal beliefs and thoughts.', 'their character.']\",\"question\":\"Nagel claims that prior to reflection, it is plausible that people cannot be morally assessed for:\",\"src\":\"ori_mmlu-philosophy\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}