{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"bc760e84-4129-57f5-a01f-9b1f32ed9abf","task_key":"default--test--1184","task_revision_id":"1","upstream_id":"1184","short_description":"One evening, a husband received a telephone call from an unidentified source who…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"[\\\"guilty of assault on his friend's wife only.\\\", \\\"guilty of attempted murder of both his friend and his friend's wife.\\\", \\\"guilty of reckless endangerment of both his friend and his friend's wife.\\\", \\\"guilty of attempted murder of his friend's wife only.\\\", 'guilty of attempted murder of his friend only.', \\\"guilty of attempted murder of his friend and assault on his friend's wife.\\\", \\\"guilty of attempted manslaughter of both his friend and his friend's wife.\\\", 'guilty of attempted manslaughter of his friend only.', \\\"not guilty of attempted murder of either his friend or his friend's wife.\\\", 'not guilty due to lack of intent to kill.']\",\"question\":\"One evening, a husband received a telephone call from an unidentified source who informed him that his wife was having an affair with his friend. As a result, the husband drove to his friend's house with a loaded shotgun. Believing that his friend was in an upstairs bedroom, the husband fired the shotgun through the dining room window, as he only wanted to teach him a lesson. Unbeknownst to the husband, his friend's wife was in the dining room and suffered a minor gunshot wound. The husband should be found\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}