{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"0969ec38-a7f7-5116-825c-150ce5499b41","task_key":"default--test--1114","task_revision_id":"1","upstream_id":"1114","short_description":"A defendant hated a victim and decided to kill him. The defendant put a loaded…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['voluntary manslaughter.', 'first-degree murder.', 'second-degree murder.', 'criminal negligence.', 'involuntary manslaughter.', 'negligent homicide.', 'justifiable homicide.', 'aggravated assault.', 'not guilty due to lack of intent.', 'not guilty due to intoxication.']\",\"question\":\"A defendant hated a victim and decided to kill him. The defendant put a loaded gun in his coat pocket and went outside for a walk, not sure if and when he may see the victim. The defendant passed by a bar and decided to go inside for a drink. While in the bar, he drank several bottles of beer and became intoxicated. While walking home from the bar, the defendant bumped into the victim. As soon as the defendant recognized him, he pulled out his gun and shot and killed the victim. In this jurisdiction, first-degree murder is defined as a premeditated and deliberate killing. All other forms of murder are second-degree murder. The defendant is guilty of\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}