{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"9ffec3cc-9a08-538a-afe4-593bffcc9a38","task_key":"default--test--1842","task_revision_id":"1","upstream_id":"1842","short_description":"A man became ill while at work and decided to go home early. When he entered his…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['Only murder.', 'Murder and involuntary manslaughter.', 'Murder, voluntary manslaughter, and involuntary manslaughter.', 'Murder and voluntary manslaughter.', 'Murder, voluntary manslaughter, involuntary manslaughter, and justifiable homicide.', 'Voluntary manslaughter and involuntary manslaughter.', 'Only voluntary manslaughter.', 'Voluntary manslaughter and justifiable homicide.', 'Murder, voluntary manslaughter, and justifiable homicide.', 'Only involuntary manslaughter.']\",\"question\":\"A man became ill while at work and decided to go home early. When he entered his bedroom, the man saw his wife engaged in sexual intercourse with a neighbor. The man grabbed a gun from a dresser drawer and shot and killed the neighbor. He was later charged and prosecuted. In a jurisdiction that follows the common law for homicide offenses, which crimes should the court instruct the jury on?\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}