{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"8d45e807-f3c1-550d-8745-9bab8d2a5c06","task_key":"default--test--1264","task_revision_id":"1","upstream_id":"1264","short_description":"One night, a victim was in her home when she heard the doorbell ring. When the…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['Assault only.', 'Kidnapping, assault, and robbery.', 'Kidnapping and larceny.', 'Robbery only.', 'Assault, battery, and robbery.', 'Robbery and burglary.', 'Kidnapping, larceny, and burglary.', 'Larceny and assault.', 'Larceny, robbery, and burglary.', 'Assault, battery, larceny, and burglary.']\",\"question\":\"One night, a victim was in her home when she heard the doorbell ring. When the victim opened the door, she was confronted by three defendants. They pushed the victim inside her house and threatened her with bodily harm if she didn't cooperate. After tying her up with ropes, the trio then proceeded to ransack the victim's home. They then placed many of the victim's valuables and other possessions into a large sack and hurriedly left her house. What crimes should the defendants be convicted of?\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}