{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"8d9de842-d813-5adf-a189-f584585ce215","task_key":"all--test--8d9de842-d813-5adf-a189-f584585ce215","task_revision_id":"1","upstream_id":"","short_description":"A woman entered a jewelry store and asked the store's owner if he had any…","config":"all","split":"test","body":"{\"choices\":\"['guilty, because it is not relevant that she returned the bracelet to the display tray.', 'guilty, because the employee had actually seen her place the bracelet in her pocketbook.', 'not guilty, because she returned the bracelet to the display tray.', \\\"not guilty, because she didn't leave the store with the bracelet in her possession.\\\"]\",\"question\":\"A woman entered a jewelry store and asked the store's owner if he had any bracelets with turquoise and mother-of-pearl inlay. The owner answered affirmatively and showed the woman two display trays of bracelets. As the woman was looking at a few of the pieces, the telephone in the store began to ring. The owner excused himself and walked to the rear of the store where he answered the telephone call. While the owner was speaking on the phone, the woman placed one of the bracelets in her pocketbook and walked a few feet toward the front door of the store. She was about to leave the store, without paying for the bracelet, when she suddenly noticed one of the other employees. Thinking that the employee had seen her place the bracelet in her pocketbook, the woman walked back to the counter and returned the bracelet to the display tray. In fact, the employee had seen the woman take the bracelet but decided not to say anything after she put it back. If the woman is subsequently prosecuted for larceny of the bracelet, she will most likely be found\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}