{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"9600f539-c3e4-597f-86c2-d8f96d31abce","task_key":"all--test--9600f539-c3e4-597f-86c2-d8f96d31abce","task_revision_id":"1","upstream_id":"","short_description":"A man, a woman, and their son went to their neighbor's house. The man intended…","config":"all","split":"test","body":"{\"choices\":\"['The man, the woman, and the son.', 'The man and the woman.', 'The woman and the son.', 'None.']\",\"question\":\"A man, a woman, and their son went to their neighbor's house. The man intended to take back some tools that he believed were his and that the neighbor was keeping unlawfully. The woman believed that the tools were the man's, and she intended to help the man take them. When the son learned that the man and the woman were going to break into the neighbor's home, he decided to accompany them. The son planned to find some items inside that might be worth taking. Arriving at the neighbor's home, the man opened the front door, which was closed but unlocked. Upon entering, the son went to the neighbor's upstairs bedroom and found a watch, which he took. In the meantime, the man and the woman went to the garage and began rummaging through the neighbor's tools. The man found the tools, which he seized. The three of them then left the neighbor's home. In this jurisdiction, burglary is defined as the breaking and entering of any structure with the intent to commit a felony therein. Which, if any, individuals should be found guilty of conspiracy?\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}