{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"7827bf4a-57cc-5918-8756-63fa63b96194","task_key":"all--test--7827bf4a-57cc-5918-8756-63fa63b96194","task_revision_id":"1","upstream_id":"","short_description":"A plaintiff alleges in a lawsuit against a store that he purchased a hot dog…","config":"all","split":"test","body":"{\"choices\":\"['inadmissible, because of the best evidence rule.', 'inadmissible, because they are irrelevant to the present claim.', 'inadmissible, because there is the danger of undue prejudice to the plaintiff, which outweighs the probative value of the evidence.', 'admissible, because they establish a pattern of similar actions.']\",\"question\":\"A plaintiff alleges in a lawsuit against a store that he purchased a hot dog with chili that gave him food poisoning and caused him excruciating pain. The store offers into evidence certified copies of court proceedings revealing that the plaintiff has filed three similar lawsuits in the last 18 months against other convenience stores, claiming food poisoning resulting from the purchase of food from those stores. These copies should be ruled\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}