{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"567e4171-885b-5c99-bec0-57e15dfe9b14","task_key":"all--test--567e4171-885b-5c99-bec0-57e15dfe9b14","task_revision_id":"1","upstream_id":"","short_description":"A defendant and his college roommate went home together for the holiday…","config":"all","split":"test","body":"{\"choices\":\"[\\\"the defendant didn't know what was in the package.\\\", \\\"the roommate didn't tell him he was carrying illegal narcotics.\\\", \\\"the package didn't belong to him.\\\", 'the cocaine was illegally seized.']\",\"question\":\"A defendant and his college roommate went home together for the holiday vacation. The defendant was to go back to college by himself. The day that the defendant was to drive back to college, his roommate asked him if he would deliver a package to one of their classmates. The defendant agreed, and the roommate gave him a small package, which the defendant placed in his glove compartment. While driving back to school, the defendant was stopped for speeding. When the state trooper asked the defendant for his license and registration, he reached into the glove compartment. As he did, the package that his roommate had given him fell onto the floor. When it hit the floor, the box broke open and a plastic envelope containing a white substance slid out. Seeing that the substance appeared to be cocaine, the state trooper arrested the defendant for possession of illegal narcotics under a state statute making it a felony to willfully possess a controlled substance. The package did, in fact, contain cocaine. If the defendant is later brought to trial for this charge, his best defense is that\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}