{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"f52eb984-a28c-5a19-8ac1-8f8dd556141f","task_key":"all--test--f52eb984-a28c-5a19-8ac1-8f8dd556141f","task_revision_id":"1","upstream_id":"","short_description":"A woman owned land that abutted a public highway to the south. A neighbor owned…","config":"all","split":"test","body":"{\"choices\":\"['No, because the neighbor has other means of access.', 'No, because the zoning code permits the use of solar collectors.', 'Yes, because the easement remains valid.', 'Yes, because the woman has other power resources.']\",\"question\":\"A woman owned land that abutted a public highway to the south. A neighbor owned the land immediately to the north. The neighbor's predecessor had received an easement from the woman's predecessor in title to cross the woman's land for access to the public highway. This access was desired even though the neighbor and his predecessor had other access to public roads. Recently, the woman erected a large solar collector on a portion of her land subject to the easement, even though the woman has other power resources. The location was essential to acquire the maximum sunlight. However, by erecting the collector there, the woman effectively cut off the neighbor's ability to use the easement. The local zoning code permits the use of solar collectors. The neighbor immediately sued to compel the woman to remove the solar collector. Must the solar collector be removed?\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}