{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"ccc76fe5-b04e-5160-86c0-e254dc7e6bcb","task_key":"all--test--ccc76fe5-b04e-5160-86c0-e254dc7e6bcb","task_revision_id":"1","upstream_id":"","short_description":"A landlord was the owner of a two-story dwelling house and leased it completely…","config":"all","split":"test","body":"{\"choices\":\"['recover, because pnvity of estate exists between a landlord and sublessee.', 'recover, because there is privity of contract between a landlord and sublessee.', 'recover, because there is both privity of estate and privity of contract between a landlord and sublessee.', 'not recover, because privity of estate does not exist between the landlord and sublessee.']\",\"question\":\"A landlord was the owner of a two-story dwelling house and leased it completely furnished to a tenant for a 10-year period. Two years after the tenant entered into the leasehold agreement with the landlord, he subleases the house to his cousin. Thereafter, the cousin defaults on the rental payments. If the landlord brings an action to recover the past rent due from the cousin, the landlord will:\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}