{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"c4aa3f3c-12ec-53ac-82d2-aa62eb4b0f1e","task_key":"all--test--c4aa3f3c-12ec-53ac-82d2-aa62eb4b0f1e","task_revision_id":"1","upstream_id":"","short_description":"A student was a practical joker who liked to perform zany antics. Late at night…","config":"all","split":"test","body":"{\"choices\":\"['Burglary only.', 'Attempted violation of the statute.', 'Both burglary and attempted violation of the statute.', 'Neither burglary nor attempted violation of the statute.']\",\"question\":\"A student was a practical joker who liked to perform zany antics. Late at night after studying, he would often walk around the campus dressed only in a raincoat and boxers. As he approached young women, he would flash them by opening his raincoat and exposing himself in his shorts. The student believed that he was committing a crime by flashing himself in front of the co-eds. In this jurisdiction, it is a felony to flash or expose oneself in the nude, so, unknown to the student, what he was doing was not a crime. One night, the student broke into a sorority house intending to flash the sorority sisters. Clad only in his raincoat and shorts, he entered the bedroom of one of the sisters who was lying in her bed studying for an exam. The sister, who knew the student, said, \\\"What are you doing here? Shouldn't you be studying for finals?\\\" The student then took off his raincoat and responded, \\\"Study this, baby!\\\" He then began to run through the sorority house in his shorts. Shortly thereafter, the police arrived and placed the student under arrest. If the student is prosecuted, he should be found guilty of which, if any, of the following crimes?\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}