{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"76a7210e-09d3-5a7a-8e09-855746538b2e","task_key":"all--test--76a7210e-09d3-5a7a-8e09-855746538b2e","task_revision_id":"1","upstream_id":"","short_description":"A large man with menacing tattoos all over his face and bald head followed…","config":"all","split":"test","body":"{\"choices\":\"[\\\"No, the large man's songs could have been coincidental or unthinking, he made no threats, and later could have been trying to return the wallet.\\\", 'Yes, because the large man followed too closely for too long, he sang menacing songs that placed the businessman in fear, and then he took the wallet and ran in the other direction.', \\\"Yes, because one should never pick up someone else's wallet laying on the ground for any reason.\\\", 'No, because the wallet was not in the immediate presence of the victim when the large man picked it up.']\",\"question\":\"A large man with menacing tattoos all over his face and bald head followed closely behind a businessman who was lost in a long dark alley. The large man followed the businessman for several blocks of unending, unlit alleys. The businessman was in great fear. The large man was just a few feet behind, singing songs about how he was going to have a \\\"big meal tonight\\\" and it looked like he \\\"had struck it rich,\\\" and other words indicating possible violence and robbery. Finally, the businessman threw his wallet in one direction and ran the other way, as he yelled, \\\"You can have my money, just leave me my life!\\\" The large man picked up the wallet and ran in the opposite direction, but when he came to the end of the alley he was arrested and charged with robbery. He appealed his conviction, arguing that he had no intent to steal and was trying to return the wallet to the victim. Will the appellate court likely affirm the robbery conviction?\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}