{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"364e611c-4942-5260-9544-ecf4f5eeec26","task_key":"default--test--1462","task_revision_id":"1","upstream_id":"1462","short_description":"A mother hated a girl because she always seemed to outperform her daughter and…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['Conspiracy and voluntary manslaughter.', 'Solicitation, attempted murder, and one count of murder.', 'Conspiracy, attempted murder, and two counts of murder.', 'Conspiracy, attempted murder, and voluntary manslaughter.', 'Solicitation and attempted murder.', 'Solicitation and two counts of murder.', 'Solicitation, conspiracy, and voluntary manslaughter.', 'Solicitation, attempted murder, and two counts of murder.', 'Conspiracy and two counts of murder.', 'Conspiracy, voluntary manslaughter, and two counts of murder.']\",\"question\":\"A mother hated a girl because she always seemed to outperform her daughter and make her feel inferior. Fearing that the girl would beat out her daughter for the last cheerleading position, the mother decided to kill the girl. One night while the mother and her boyfriend were having dinner, the mother asked him to kill the girl. The mother handed her boyfriend a gun and gave him the home address where the girl lived. Unknown to the boyfriend, the mother gave him the wrong address. By mistake, the boyfriend went to the home of a family who lived across the street from the girl. The boyfriend rang the doorbell and a woman opened the door. The boyfriend asked the woman if her daughter was home. The woman called for her daughter to come to the front door. When she did so, the boyfriend shot and killed both the woman and her daughter. The mother should be found guilty for which, if any, of the following crimes?\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}