{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"db838faf-ee6d-5eb1-808f-a775d516d752","task_key":"all--test--db838faf-ee6d-5eb1-808f-a775d516d752","task_revision_id":"1","upstream_id":"","short_description":"A business owner had a feud and \"bitter blood\" with a former employee. One day…","config":"all","split":"test","body":"{\"choices\":\"['No, because she acted in self-defense.', 'No, because these facts constitute justifiable homicide.', 'Yes, because she had been lying in wait for the victim.', 'Yes, because the owner cannot claim self-defense when her motive is evil.']\",\"question\":\"A business owner had a feud and \\\"bitter blood\\\" with a former employee. One day that employee came uninvited into the owner's office brandishing a loaded shotgun, approached the owner's desk and raised the weapon. The owner, however, was secretly waiting for the opportunity. As the employee began to press on the trigger, the owner pulled a gun from her drawer and killed him. The owner admitted she felt malice toward her enemy. The authorities charged her with murder. Can she be convicted?\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}