{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"93f0bac7-a375-5267-8961-86fdbd2483ea","task_key":"all--test--93f0bac7-a375-5267-8961-86fdbd2483ea","task_revision_id":"1","upstream_id":"","short_description":"A homeowner was holding his annual party at his home. His parties attracted a…","config":"all","split":"test","body":"{\"choices\":\"['guilty, because he crashed the party.', 'guilty, because he intended to steal the baseball.', 'not guilty, because he crashed the party only to have a good time.', 'not guilty, because the front door was unlocked.']\",\"question\":\"A homeowner was holding his annual party at his home. His parties attracted a wide variety of diverse individuals. On the night in question, a defendant was loafing at the local bar when a number of the invited guests came in to purchase spirits for the party. The defendant decided to crash the party. He arrived at the homeowner's house around midnight when the party was in full swing. He went in through the front door, which was unlocked, and easily mingled with the other partygoers. During the party, the defendant wandered through the house and came upon a baseball, signed by a famous baseball player, in a glass case. Knowing the ball to be very valuable, the defendant put it under his jacket and made his way down the stairs and out the front door. If the defendant is subsequently charged with burglary, he will probably be found\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}