{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"c51e29dc-1ce3-5ca4-8a00-3431496d4fb9","task_key":"all--test--c51e29dc-1ce3-5ca4-8a00-3431496d4fb9","task_revision_id":"1","upstream_id":"","short_description":"A man suffered frostbite when he spilled a soft drink on his lap that he had…","config":"all","split":"test","body":"{\"choices\":\"[\\\"admissible to prove the restaurant's negligence, provided that the court gives a cautionary instruction that the testimony should not be considered as bearing on the issue of damages.\\\", 'admissible to prove that the restaurant should have been aware of the danger of frostbite posed by its soft drinks.', 'inadmissible, because it is not probative of a fact in issue.', 'inadmissible, because it seeks to put into evidence separate, unrelated transactions with a third party.']\",\"question\":\"A man suffered frostbite when he spilled a soft drink on his lap that he had purchased from a restaurant. The man sued the restaurant for negligence and damages. At trial, the man's attorney calls a witness to testify that the witness also suffered frostbite when she spilled a soft drink in her lap that she had purchased at the restaurant two weeks earlier. The witness also states that she told the restaurant about her injury immediately after suffering it. Upon objection by the restaurant's attorney, the trial judge should rule the witness's testimony\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}