{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"23904207-4918-542a-91e3-11a509cb6a90","task_key":"default--test--1343","task_revision_id":"1","upstream_id":"1343","short_description":"An off-duty police officer was standing on a street corner waiting for a bus. A…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['murder.', 'negligent homicide.', 'reckless endangerment.', 'no crime, it was self-defense.', 'aggravated assault.', 'voluntary manslaughter.', 'justifiable homicide.', 'attempted robbery.', 'assault with a deadly weapon.']\",\"question\":\"An off-duty police officer was standing on a street corner waiting for a bus. A man came up from behind and stole the police officer's wallet from his pants pocket. As the man was running away with the wallet, the police officer pulled out his service revolver. The police officer yelled at the man to stop and then fired several shots in the man's direction. The police officer did not aim directly at the man but shot at the pavement intending to frighten him. One of the bullets ricocheted off the sidewalk and struck the man, killing him. The police officer is guilty of\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}