{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"a641e49e-2cc2-5d83-b4b6-7e48786c6631","task_key":"all--test--a641e49e-2cc2-5d83-b4b6-7e48786c6631","task_revision_id":"1","upstream_id":"","short_description":"Sally Sobriety, owner of Greenacre and Blackacre, was a recovering alcoholic.…","config":"all","split":"test","body":"{\"choices\":\"['Betty // Betty', 'Friend // Betty', 'Betty // Associate', 'Friend // Associate']\",\"question\":\"Sally Sobriety, owner of Greenacre and Blackacre, was a recovering alcoholic. Sally's only heir was her daughter Betty Boozer, a heavy drinker. In an attempt to encourage Betty to go on the wagon, Sally conveyed Greenacre to \\\"Friend and her heirs, but if liquor is ever consumed on the property, then to Betty.\\\" Sally conveyed Blackacre to \\\"Associate and her heirs, but if Associate ever consumes alcohol on the property, then to Betty.\\\" One year later, Friend and Associate both began hosting wine-tasting events on their properties, at which they each personally consumed substantial alcohol. Sally was so shocked that she had a stroke and died. The property interests of the following parcels of land should go to\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}