{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"28494bef-ad57-50d3-9bc2-8d8adf3ed1fe","task_key":"default--test--898","task_revision_id":"1","upstream_id":"898","short_description":"A defendant and a victim were roommates at college. The defendant was playing…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['guilty of attempted murder.', 'guilty of manslaughter.', 'not guilty.', 'guilty of battery.', 'guilty of negligent homicide.', 'guilty of murder.', 'guilty of reckless endangerment.', 'guilty of second-degree murder.', 'guilty of first-degree murder.', 'guilty of aggravated assault.']\",\"question\":\"A defendant and a victim were roommates at college. The defendant was playing the role of a serial killer in the school play and asked the victim if she could leave their apartment for a few hours while she rehearsed. The victim agreed but returned to the apartment shortly thereafter because she was curious and wanted to see the defendant rehearse. The victim quietly came back into the apartment and hid behind some curtains in the dining room in order to watch the defendant rehearse. In her role, the defendant earned a large butcher's knife, with which she stabbed the curtains at her \\\"victim. \\\" Unknown to the defendant, the victim, who was hiding behind the curtain, was stabbed in the chest and killed. If the defendant is prosecuted for the victim's death, she should be found\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}