{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"387d576c-9fb0-5227-8092-d16dbae06889","task_key":"all--test--387d576c-9fb0-5227-8092-d16dbae06889","task_revision_id":"1","upstream_id":"","short_description":"A widow owned an apartment building. Several years ago, a student, in a signed…","config":"all","split":"test","body":"{\"choices\":\"['the leasehold contract was void ab initio.', 'the widow had not given the student requisitenotice for terminating a periodic tenancy.', 'the widow had not given the student requisitenotice for terminating a term of years.', 'the terms of the leasehold contract were binding on both parties.']\",\"question\":\"A widow owned an apartment building. Several years ago, a student, in a signed writing, entered a three-year lease agreement with the widow. The student agreed to pay $800 monthly rent for the third-floor apartment in the widow's building. The original three-year term had long since expired, but the student had continued to pay his rent, and the widow continued to accept it. If the widow had chosen to evict the student immediately upon the end of the three-year period, and before she had accepted the next rent check, the student's strongest argument to prevent eviction, if true, would have been that\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}