{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"6e4d7275-5238-5701-8a26-309f19314136","task_key":"all--test--6e4d7275-5238-5701-8a26-309f19314136","task_revision_id":"1","upstream_id":"","short_description":"A defendant was employed by a railroad company as a watchman at its crossing, to…","config":"all","split":"test","body":"{\"choices\":\"['granted, because the police officer failed to give the defendant his Miranda warnings.', \\\"granted, because the police officer's conduct in questioning the defendant immediately after he awoke was unfairly prejudicial to the defendant.\\\", 'denied, because the exchange took place in a non-custodial setting and the question was investigatory in nature.', \\\"denied, because the defendant's incriminating statements were voluntary.\\\"]\",\"question\":\"A defendant was employed by a railroad company as a watchman at its crossing, to give warning to the public of approaching trains. Late one evening, he fell asleep in his kiosk and failed to warn of the approach of an oncoming train. A man, who was driving his car, knew of the usual presence of the watchman. As he approached the crossing, he received no warning. Driving onto the track, the man's car was struck and crushed by the train. The man died instantly. A few minutes after the accident, a police officer arrived at the crossing and walked into the kiosk. As he entered, the defendant then awoke. The police officer asked him, \\\"What happened here?\\\" The defendant made incriminating statements. The defendant is subsequently charged with involuntary manslaughter. His motion to prevent the introduction of his incriminating statements into evidence will most likely be\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}