{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"16dc3baf-453c-580a-89a4-b88294a80be6","task_key":"default--test--11129","task_revision_id":"1","upstream_id":"11129","short_description":"One message that Nathanson argues we can send by abolishing the death penalty is…","config":"default","split":"test","body":"{\"category\":\"philosophy\",\"options\":\"['we believe in the possibility of redemption and change.', 'there are some penalties worse than death.', 'we are unable to administer justice effectively.', 'there is no punishment that is proportional to murder.', 'there is always a possibility of judicial errors.', 'we are lenient towards heinous crimes.', 'we should prioritize rehabilitation over punishment.', 'none of the above', 'we respect the dignity of human life.', 'we have a flawed criminal justice system.']\",\"question\":\" One message that Nathanson argues we can send by abolishing the death penalty is that\",\"src\":\"ori_mmlu-moral_disputes\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}