{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"ae7a52f0-c3de-59e2-9f43-345540fefdb8","task_key":"default--test--1728","task_revision_id":"1","upstream_id":"1728","short_description":"An architect is the owner of a large two-story office building. The architect…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"[\\\"lose, because the obligation to pay rent is dependent on the architect's performance of his express covenant.\\\", 'win, because the tenant left the premises before the lease term was over', 'win, because the cost of repainting is less than the rent due', \\\"lose, because the tenant was justified in leaving due to the architect's neglect of his duties.\\\", 'n, because the tenant had the remedy of self-help.', 'win, because there was no constructive eviction.', 'win, because the tenant did not take legal action against the architect for failure to repaint', 'lose, because the tenant has the right to withhold rent until the repainting is done', 'lose, because he cannot maintain an action for rent while in breach of an express covenant.']\",\"question\":\"An architect is the owner of a large two-story office building. The architect designed and supervised the building's construction. He and his associate occupy the first floor for their architectural business. In June 2005, the architect entered into a valid and binding written lease with a tenant to rent the second floor office space for four years at a monthly rental of $5,000. The lease contained a provision wherein the architect was required to repaint the second floor premises after two years of occupancy. On June 1, 2007, the tenant sent the architect a written letter requesting the repainting of the second-floor office space. The architect made no reply and failed to perform the repainting as the lease required. The cost of repainting the tenant's office was estimated at $1,750. On July 1, 2008, the architect had still not repainted the premises. The following day, the tenant moved out, mailed the keys to the architect, and refused to pay any more rent. The lease was silent as to the rights and remedies of the tenant due to the architect's failure to perform the repainting. There is no applicable statute in the jurisdiction. In an appropriate action by the architect against the tenant for the rent due, the architect will\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}