{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"9743e5dd-73dc-5229-81eb-1c48b32f47be","task_key":"all--test--9743e5dd-73dc-5229-81eb-1c48b32f47be","task_revision_id":"1","upstream_id":"","short_description":"Under which of the following situations would the defendant most likely be found…","config":"all","split":"test","body":"{\"choices\":\"['A defendant conspired with two other men to rob a store. On the day of the robbery, the three participants drove to the store in a stolen van. In accordance with their plan, the defendant was to wait in the getaway van while the two men conducted their heist. While the two men were inside the store, the defendant decided to renounce his involvement and participation in the scheme and hurriedly drove away in the van. In his haste, the defendant failed to see a pedesthan jaywalking across the street, and struck and killed the pedestrian.', 'A defendant, accompanied by his girlfriend, was attending his law school graduation party. Both the defendant and his girlfriend became highly intoxicated during the party celebration. As the defendant and his girlfriend were leaving the party, the defendant handed his girlfriend the keys to his automobile and directed her to drive home. While driving, the girlfriend suddenly swerved the car across the median strip into the path of an oncoming car, killing the driver of the other vehicle.', \\\"A defendant, whose driver's license had been revoked one week previously, was driving on an infrequently traveled country road one night. Knowing that his car brakes were defective, the defendant was traveling at 35 m. p. h. in a zone in which the speed limit was 25 m. p. h. when a victim suddenly darted in front of his car. In an effort to avoid striking the victim, the defendant applied his brakes to no avail. The car struck and killed the victim.\\\", 'One afternoon a defendant was delivering pornographic materials to various adult bookstores in the community. In this jurisdiction, the delivery of pornographic materials constituted a misdemeanor. As the defendant was on his way to a book store for his final delivery of the day, he inadvertently struck a blind man who was crossing an intersection. Six months later, the blind man died as a result of injuries suffered in the accident.']\",\"question\":\"Under which of the following situations would the defendant most likely be found guilty of murder?\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}