{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"dc434966-87d5-5345-b946-511dc2de4000","task_key":"all--test--dc434966-87d5-5345-b946-511dc2de4000","task_revision_id":"1","upstream_id":"","short_description":"A man was at a party where he observed a female guest who had passed out and was…","config":"all","split":"test","body":"{\"choices\":\"[\\\"Yes, robbery is the unlawful taking of property from another's person by intimidation or force.\\\", 'No, this was not robbery because there was no intimidation, and indeed no force, against an unconscious victim.', 'Yes, robbery does not require that the victim be subdued by force or be placed in fear.', 'No, this was not robbery because it occurred in a private home and not in a business establishment or on the street.']\",\"question\":\"A man was at a party where he observed a female guest who had passed out and was unconscious and alone in a bedroom of the house. The man locked the door and quietly took the female's purse, emptied all of the money into his pockets, and removed the jewelry from her person. He was apprehended and charged with robbery. Can he be convicted of that crime?\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}