{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"dff12d18-d325-5fc8-9f75-63d10df3855f","task_key":"default--test--11001","task_revision_id":"1","upstream_id":"11001","short_description":"According to Velleman, who, if anyone, should decide whether a patient receives…","config":"default","split":"test","body":"{\"category\":\"philosophy\",\"options\":\"['a court of law', 'the hospital administration', 'no one, it should not be allowed under any circumstance', \\\"the patient's caregiver\\\", 'a medical ethics committee', \\\"the patient's family\\\", \\\"the patient's physician\\\", \\\"the patient's legal representative\\\", 'legislators', 'the patient']\",\"question\":\" According to Velleman, who, if anyone, should decide whether a patient receives euthanasia?\",\"src\":\"ori_mmlu-moral_disputes\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}