{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"cd96ae82-d1db-5fff-a1f4-4ec5b7ad01b9","task_key":"all--test--cd96ae82-d1db-5fff-a1f4-4ec5b7ad01b9","task_revision_id":"1","upstream_id":"","short_description":"A police officer observed two men acting suspiciously in front of a convenience…","config":"all","split":"test","body":"{\"choices\":\"['Yes, because there was no probable cause to believe that a crime was being committed.', 'No, because the officer had reasonable suspicion to believe that there might be criminal activity afoot.', 'No, because the officer did have probable cause to believe that a crime was being attempted.', \\\"Yes, because the officer is not allowed to take anything out of the men's pockets while conducting a pat-down.\\\"]\",\"question\":\"A police officer observed two men acting suspiciously in front of a convenience store. They paced back and forth, walked around the store, and periodically looked in the window. The experienced officer feared that these individuals were most likely \\\"casing\\\" the place prior to robbing it. He approached, identified himself, and patted down the outer clothing of each. He found a hand gun in each man's pocket and arrested them for carrying concealed weapons. Will the court suppress the guns on the basis that there was no probable cause and the search was unconstitutional?\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}