{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"95eec84f-009c-5800-99c9-2389ef927b28","task_key":"all--test--95eec84f-009c-5800-99c9-2389ef927b28","task_revision_id":"1","upstream_id":"","short_description":"While on vacation, two friends went scuba diving in the ocean. Each of them was…","config":"all","split":"test","body":"{\"choices\":\"['in favor of both defendants if no evidence is presented showing who actually shot the spear that injured the victim.', 'against both defendants jointly unless one of them proves that he did not shoot the spear that struck the victim.', 'against each defendant for one-half of the amount of damages, because they both acted independently and not jointly.', 'against each defendant for one-half of the amount of damages, because they both shot their spear guns simultaneously.']\",\"question\":\"While on vacation, two friends went scuba diving in the ocean. Each of them was equipped with a spear gun, which they planned to use to kill tuna. As they were swimming, both men saw what appeared to be a large fin entangled in a nearby coral reef. Simultaneously, they shot their spear guns at the projecting wing-like figure. Unbeknownst to either of the friends, it was not a fish but rather another scuba diver. The victim was struck and slightly wounded by one of the spears. The victim asserts a claim for damages against the two friends. At trial, the judge should instruct the jury to rule\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}