{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"79fe5947-6839-553b-baeb-9f3341bc5cea","task_key":"default--test--1249","task_revision_id":"1","upstream_id":"1249","short_description":"Defendant was waiting in line for post-Thanksgiving day sales. The door opened…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['Battery', 'First degree murder', 'Premeditated murder', 'Involuntary manslaughter', 'Second degree murder', 'Aggravated assault', 'Third degree murder', 'No crime, it was a accident.', 'voluntary manslaughter']\",\"question\":\"Defendant was waiting in line for post-Thanksgiving day sales. The door opened at 6 a.m. and hundreds of people rushed the entrance for a chance at a hugely discounted holiday gift. When defendant saw that there was a signed copy of a book written by his mother's favorite author, he jumped towards it, only to be blocked by a little old lady with a baseball bat. She was using the bat as a blocking tool, but he was determined to get the book, so he deliberately stuck his leg out in front of her and tripped her. She fell hard and in the process ended up with a blood clot in her leg, which traveled to her heart over a period of weeks, and ultimately caused her death. What crime can the defendant likely be convicted of?\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}