{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"92b5c26b-fb47-5e14-a259-d8e08855ff36","task_key":"default--test--1166","task_revision_id":"1","upstream_id":"1166","short_description":"A new gang member, hoping to impress the gang's leader, pointed a gun at a…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['Accomplice to robbery.', 'Accessory before the fact to robbery.', 'Coercion.', 'No crime, as he did not directly participate in the robbery.', 'Theft by receiving.', 'Accessory after the fact to robbery.', 'Aggravated assault.', 'Grand larceny.', 'Conspiracy to commit robbery.']\",\"question\":\"A new gang member, hoping to impress the gang's leader, pointed a gun at a pedestrian and ordered her to give him her expensive watch, which she did. The gang member then tossed the watch to the gang leader, who was standing nearby. Although totally surprised by this act, the gang leader put the watch in his pocket. The pedestrian ran away. What crime did the gang leader commit?\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}