{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"17db3fc7-8f4e-57fb-84da-b13b00349f9a","task_key":"default--test--1888","task_revision_id":"1","upstream_id":"1888","short_description":"A husband came home early from work one day. He walked to the bedroom and, upon…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['No crime.', 'Second-degree murder.', 'Murder.', 'Self-defense.', 'First-degree murder.', 'Involuntary manslaughter.', 'Assault.', 'Justifiable homicide.', 'Aggravated assault.']\",\"question\":\"A husband came home early from work one day. He walked to the bedroom and, upon opening the door, saw his wife and another man in bed together. Visibly upset, the husband told the other man to leave. As the man was reaching for his pants, he pulled out a gun, fearing that the husband was about to attack him. When the husband saw the gun, he jumped on the man and angrily twisted his neck, breaking it and killing him. The husband is guilty for which, if any, of the following crimes?\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}