{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"8dd5a619-ccc1-5ee3-a7ab-d15374b2da9f","task_key":"all--test--8dd5a619-ccc1-5ee3-a7ab-d15374b2da9f","task_revision_id":"1","upstream_id":"","short_description":"A writer owned a building in a city. This was the only piece of real estate that…","config":"all","split":"test","body":"{\"choices\":\"['created a periodic tenancy.', 'could be terminated at will by either party.', 'did not convey to the shopkeeper a term of years at law.', 'was void, invalid, and had no legal effect.']\",\"question\":\"A writer owned a building in a city. This was the only piece of real estate that the writer owned. The three-story building had a store on the ground floor and apartments on the other two floors. The writer entered into a leasehold agreement with a shopkeeper, who would lease the first floor, where she planned to open a sporting goods store. After identifying the parties, the operative words of the lease were as follows: \\\"Landlord hereby agrees to lease for the three years the first floor of his building in the city to tenant, reserving unto said landlord annual rental in the sum of $12,000 payable in advance in monthly installments of $1,000. \\\"At the moment of signing, the leasehold agreement entered into between the writer and the shopkeeper\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}