{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"28c004bc-f1c0-57aa-93b0-c030bb954ebd","task_key":"default--test--1478","task_revision_id":"1","upstream_id":"1478","short_description":"A defendant, who was an indigent, was arrested and charged with possession of…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['One year in jail, but with the possibility of parole after six months.', 'Two years in jail.', 'Three months in jail and a $250 fine.', 'Probation for one year and a $500 fine.', 'A $500 fine.', 'Six months in jail.', 'Community service for one year.', 'One year in jail.']\",\"question\":\"A defendant, who was an indigent, was arrested and charged with possession of less than one ounce of marijuana. Under the relevant state statute, possession of less than one ounce of marijuana is a misdemeanor, punishable by a maximum of one year in jail and/or a fine of $500. At trial, the defendant pleaded not guilty and requested that an attorney be appointed to represent him. The trial judge refused to honor the defendant's request for an attorney. If the defendant is subsequently found guilty, which of the following is the most severe sentence that can constitutionally be imposed?\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}