{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"a4e05ca3-28b1-50c1-9671-8f21947b27f3","task_key":"all--test--a4e05ca3-28b1-50c1-9671-8f21947b27f3","task_revision_id":"1","upstream_id":"","short_description":"A defendant was arrested at an airport when the small suitcase he was carrying…","config":"all","split":"test","body":"{\"choices\":\"['objectionable, because the defendant has a privilege against self-incrimination.', 'objectionable, because the probative value is substantially outweighed by the danger of unfair prejudice.', \\\"permissible, because such evidence is relevant to the defendant's credibility.\\\", 'permissible, because the defendant waived his privilege against self-incrimination by taking the stand.']\",\"question\":\"A defendant was arrested at an airport when the small suitcase he was carrying was found to contain heroin. The defendant, who did not challenge the legality of the airport search, was subsequently prosecuted for possession of heroin. At trial, the defendant testified on his own behalf and said that the suitcase belonged to his girlfriend who was accompanying the defendant when he was arrested. The girlfriend died in a skydiving accident two weeks before the defendant's trial. Moreover, the defendant testified that although he was a former heroin addict, he had not used any heroin in the past three years. On cross-examination, the prosecuting attorney asked the defendant to roll up the sleeves of his shirt and exhibit his arms to see if there were any needle marks. This request is\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}