{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"f03e3ebd-35cf-5da7-985b-00667cd2638d","task_key":"default--test--1794","task_revision_id":"1","upstream_id":"1794","short_description":"One year ago, while driving through an intersection, the defendant's car…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"[\\\"admitted, because the plaintiff's statement to the investigator is a prior inconsistent statement.\\\", \\\"excluded, because the plaintiff's statement to the investigator is hearsay.\\\", \\\"admitted, because the plaintiff's cell phone use is relevant to the issue of negligence.\\\", \\\"excluded, because the investigator's testimony is second-hand information and therefore unreliable.\\\", 'admitted, because the accident investigator gathered the information in the course of his official duties.', \\\"admitted, because the plaintiff's inconsistent statement has bearing on her credibility.\\\", 'admitted, because it is a statement against interest and therefore an exception to the hearsay rule.', 'excluded, because the statement was not made under oath and therefore may not be truthful.', 'excluded, because the defendant did not have the opportunity to cross-examine the plaintiff at the time she made the statement.']\",\"question\":\"One year ago, while driving through an intersection, the defendant's car collided with the plaintiff's car. The plaintiff sued, claiming that the defendant ran a red light. In her case-in-chief, the plaintiff testified that immediately after the accident, a bystander, who was a local resident, screamed at the defendant, \\\"You jerk. . . you drove through the red light. \\\" In his case-in-chief, the defendant now calls a police accident investigator to testify that the plaintiff told him that she \\\"didn't see the color of the light when she entered the intersection because she was dialing a number on her cell phone. \\\"The investigator's testimony should be\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}