{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"43270d8c-3c26-575f-adb0-9dbb869e6d50","task_key":"all--test--43270d8c-3c26-575f-adb0-9dbb869e6d50","task_revision_id":"1","upstream_id":"","short_description":"Mary, a wealthy St. Petersburg widow, executed her first and only will on May…","config":"all","split":"test","body":"{\"choices\":\"['100% to Joan.', '100% to Joan if she files a timely petition requesting that the devise to the Salvation Army be avoided.', '50% to Joan and 50% to the Salvation Army.', '50% to Joan and the income from the remaining 50% to Joan for life, remainder to the Salvation Army, if Joan files a timely petition protesting the devise to the Salvation Army.']\",\"question\":\"Mary, a wealthy St. Petersburg widow, executed her first and only will on May 15, 1990 and died on August 18, 1990. Her will provided that her estate be divided equally between her only child, Joan, and the Salvation Army of Largo. How will Mary's estate actually be distributed?\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}