{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"65519617-9c9f-5c1f-8769-ca8a72f69a63","task_key":"all--test--65519617-9c9f-5c1f-8769-ca8a72f69a63","task_revision_id":"1","upstream_id":"","short_description":"An elderly woman entered a convenience store and tried to cash her Social…","config":"all","split":"test","body":"{\"choices\":\"['No crime.', 'Robbery.', 'Attempted robbery.', 'False pretenses.']\",\"question\":\"An elderly woman entered a convenience store and tried to cash her Social Security check. The owner told the woman that he was sorry but it was his policy not to cash Social Security checks. Terribly upset, the woman reached into her pocketbook and said, \\\"Listen, buddy, I've got a bomb in here, and if you don't give me the money, I'm going to blow up this joint. \\\" Actually, the woman did not have a bomb in her possession. The owner, who was not deceived by her threat, felt sorry for the woman and agreed to cash the check. The woman, who walked with the assistance of a cane, took the money and limped out of the store. After she left, the owner noticed that the woman had not endorsed the check. The woman has committed which, if any, of the following crimes?\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}