{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"d88ef18e-1de7-59fe-b22c-561d7656a9c3","task_key":"all--test--d88ef18e-1de7-59fe-b22c-561d7656a9c3","task_revision_id":"1","upstream_id":"","short_description":"A man who owned a business believed that one of his employees was stealing…","config":"all","split":"test","body":"{\"choices\":\"['The back door to the house was unlocked.', 'The burglary was abandoned.', 'The house was not occupied at the time of his entry.', 'The man did not intend to commit a crime inside the house.']\",\"question\":\"A man who owned a business believed that one of his employees was stealing computer equipment from the business. He decided to break into the employee's house one night, when he knew that the employee and her family would be away, to try to find and retrieve the equipment. The man had brought a picklock to open the employee's back door, but when he tried the door, he found that it was unlocked, so he entered. As the man was looking around the house, he heard sounds outside and became afraid. He left the house but was arrested by police on neighborhood patrol. What is the man's strongest defense to a burglary charge?\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}