{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"6783be7d-aeea-5f78-8cd5-62930aa614a1","task_key":"all--test--6783be7d-aeea-5f78-8cd5-62930aa614a1","task_revision_id":"1","upstream_id":"","short_description":"The defendant was caught in a thunderstorm while walking down the street. As the…","config":"all","split":"test","body":"{\"choices\":\"['Battery.', 'Larceny.', 'Involuntary manslaughter.', 'No crime.']\",\"question\":\"The defendant was caught in a thunderstorm while walking down the street. As the defendant was about to open an umbrella that she was carrying, a stranger to the defendant came up to her, snatched the umbrella out of the defendant's hand, and said, \\\"You thief! That's my umbrella. \\\" Enraged by being accosted in such a manner, the defendant grabbed for the umbrella with one hand and pushed the stranger with the other. The stranger hung on to the umbrella but fell over backward onto the sidewalk, which was wet and slippery. When the defendant saw the stranger hit the ground, she calmed down, decided the umbrella was not worth all the commotion, and walked off. As it turned out, the umbrella did in fact belong to the stranger, and the defendant had picked it up by mistake when she was left a restaurant earlier that day. A few moments later, the stranger got up in a daze and stepped into the gutter, where he was struck by a car that was passing another car on the right, in violation of a state law. The stranger died in the hospital two hours later. Which of the following is the most serious crime for which the defendant could be found guilty?\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}