{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"d5bb3976-fa13-50f8-b919-50a5c75d568e","task_key":"all--test--d5bb3976-fa13-50f8-b919-50a5c75d568e","task_revision_id":"1","upstream_id":"","short_description":"An elderly woman was riding in an elevator from her apartment on the 12th floor…","config":"all","split":"test","body":"{\"choices\":\"['The woman, because the smoke touched her face.', 'The woman, because she had a reasonable basis for fearing that the man would attack her.', 'The man, because he did not touch the woman.', 'The man, because his conduct was annoying but did not inflict any bodily injury on the woman.']\",\"question\":\"An elderly woman was riding in an elevator from her apartment on the 12th floor to the lobby. When the elevator stopped on the fifth floor, a man entered the elevator smoking a cigar. The man was standing in front of the elderly woman on the elevator when the woman tapped him on the shoulder. When the man turned around, the woman pointed to the \\\"No Smoking\\\" sign and said, \\\"Excuse me, sir, would you mind putting that cigar out?\\\" The man indignantly responded by inhaling heavily on his cigar, and then he blew a big puff of smoke into the woman's face. When the elevator stopped on the next floor, the man then departed. If the woman brings a civil suit against the man for battery, who will prevail?\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}