{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"100d0004-204d-5715-9037-3ba6bf310f4b","task_key":"all--test--100d0004-204d-5715-9037-3ba6bf310f4b","task_revision_id":"1","upstream_id":"","short_description":"A man informed the police that he wanted to confess to a murder. After they read…","config":"all","split":"test","body":"{\"choices\":\"['Yes, he was insane and could not have given a voluntary confession.', 'Yes, the police overreached by not having him mentally evaluated before questioning him.', 'No, there was no police misconduct or overreaching, and as a result the confession is voluntary.', 'No, a confession is admissible when accompanied by actions by the defendant that prove the truth of the statements.']\",\"question\":\"A man informed the police that he wanted to confess to a murder. After they read him his constitutional rights, he insisted that he was acting voluntarily. He then took them to the scene of the murder where they found the victim's body. By the next day, the man claimed that \\\"voices\\\" had ordered him to confess and he didn't act voluntarily. A physician verified that the man suffered from hallucinations that impaired his volitional abilities, preventing a voluntary confession. Will the court suppress the statements?\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}