{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"d49d9190-6dce-5446-bc27-7fb4d8998011","task_key":"all--test--d49d9190-6dce-5446-bc27-7fb4d8998011","task_revision_id":"1","upstream_id":"","short_description":"A woman was employed as a sales clerk at a department store. She worked in the…","config":"all","split":"test","body":"{\"choices\":\"['attempted larceny.', 'larceny.', 'false pretenses.', 'embezzlement.']\",\"question\":\"A woman was employed as a sales clerk at a department store. She worked in the women's lingerie section. One evening around closing time, the woman took a diamond necklace from a cabinetin the jewelry section of the department store and wrapped it in tissue paper. The woman then surreptitiously stuffed the necklace in a shopping bag, intending to remove it when she left the store. Moments later, the store owner approached the woman and told her she needed to stay late in order to do inventory. While the woman was completing inventory, the owner discovered the necklace and called the police. The woman is most likely guilty of\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}