{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"ebad10f0-ff9a-5d31-9e1c-4f7ddbd56852","task_key":"default--test--11196","task_revision_id":"1","upstream_id":"11196","short_description":"The retributive theory of punishment does not claim that","config":"default","split":"test","body":"{\"category\":\"philosophy\",\"options\":\"['specific punishments are justified based on the overall intrinsic value that they produce', 'wrongdoers deserve to be punished.', 'the punishment should fit the crime.', 'the punishment should be as severe as the crime.', 'wrongdoers should be punished regardless of their personal circumstances.', 'the death penalty might be morally justified in some cases of murder.']\",\"question\":\" The retributive theory of punishment does not claim that\",\"src\":\"ori_mmlu-moral_disputes\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}