{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"40cbd3fa-f2cf-5401-b617-6fd902fb2c70","task_key":"default--test--1864","task_revision_id":"1","upstream_id":"1864","short_description":"Following their law school graduation party, a graduate and his roommate were…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['admissible under the former testimony exception to the hearsay rule.', 'inadmissible, because it is considered hearsay.', 'admissible as a declaration against interest.', 'admissible, because the graduate consented to its use.', 'inadmissible, because the roommate is not a party to the lawsuit.', 'inadmissible, because the suit was dismissed as part of a compromise.', 'admissible under the business records exception to the hearsay rule.', 'admissible as a statement of fact.', 'admissible as a prior inconsistent statement.', 'admissible as an admission.']\",\"question\":\"Following their law school graduation party, a graduate and his roommate were driving home in an automobile that was owned and operated by the graduate. As they were approaching an intersection, their vehicle collided with a motorcycle whose rider was seriously injured in the accident. The rider sued the graduate for operating his motor vehicle in excess of the speed limit. In addition, the roommate brought suit against the rider for operating his motorcycle negligently and driving through a red light. The roommate subsequently reached an out- of-court settlement with the graduate and dismissed the lawsuit against him. At trial against the rider, the roommate testified that he observed the rider drive his motorcycle through a red light and then swerve into the path of the graduate's automobile. In rebuttal, the rider offered into evidence the roommate's complaint against the graduate alleging that the latter was driving \\\"at an excessively high rate of speed at the time the accident occurred. \\\"The complaint is\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}