{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"0a995323-6821-5c8b-8bfb-cf5d444f4e27","task_key":"default--test--10781","task_revision_id":"1","upstream_id":"10781","short_description":"Identify the conclusion of the following argument. Remember to remove any…","config":"default","split":"test","body":"{\"category\":\"philosophy\",\"options\":\"['A murderer is not wrong in killing his victim.', 'If a murderer is right in killing his victim, then society is also right in killing the murderer.', 'And a murderer is wrong in killing his victim.', 'This does not follow for the reason that if a murderer is wrong in killing his victim, then society is also wrong in killing the murderer.', 'And a murderer is not wrong in killing his victim.', 'This follows for the reason that if a murderer is wrong in killing his victim, then society is also wrong in killing the murderer.', 'If a murderer is wrong in killing his victim, then society is also wrong in killing the murderer.', 'Society is not wrong in killing the murderer.', 'A murderer is right in killing his victim.', 'It is wrong for society to kill a murderer.']\",\"question\":\"Identify the conclusion of the following argument. Remember to remove any conclusion indicators. It is wrong for society to kill a murderer. This follows for the reason that if a murderer is wrong in killing his victim, then society is also wrong in killing the murderer. And a murderer is wrong in killing his victim.\",\"src\":\"ori_mmlu-formal_logic\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}