{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"d2daf7d8-522e-50e3-a15d-2361c142fa30","task_key":"all--test--d2daf7d8-522e-50e3-a15d-2361c142fa30","task_revision_id":"1","upstream_id":"","short_description":"A defendant went to a convenience store intending to rob it. The defendant had a…","config":"all","split":"test","body":"{\"choices\":\"['guilty, because the owner was placed in fear.', 'guilty, because the defendant entered the store with a gun in his possession.', 'not guilty, because the defendant did not make any threat of force.', \\\"not guilty, because the defendant did not take any money from the victim's person.\\\"]\",\"question\":\"A defendant went to a convenience store intending to rob it. The defendant had a gun inside his coat pocket. When the defendant entered the store, the owner saw that he had his hand in his coat pocket. Although the owner did not actually see the gun, he noticed a bulge in the defendant's coat pocket. Paranoid because of a rash of recent robberies, the owner said, \\\"Please don't hurt me 11 do anything you want. \\\" The owner then fainted and fell to the floor. The defendant walked behind the counter and opened the cash register. He took the money from the register and left the store. If the defendant is charged with robbery, he should be found\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}