{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"9ac77aa8-fef0-5320-a5da-fc652d597c6e","task_key":"default--test--1858","task_revision_id":"1","upstream_id":"1858","short_description":"A landlord was the owner of a two-story dwelling house and leased it completely…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['the landlord, because he failed to inspect the property after the storm', 'both the landlord and the tenant, because they both neglected their responsibilities in maintaining the property.', 'the tenant, because a tenant for years is obligated to make such ordinary repairs on the leased property.', 'the landlord, because he was under an implied obligation to keep the premises in a habitable condition.', 'the landlord, because he was under an affirmative obligation to deliver the premises in a reasonable state of repair.', 'the tenant, because he noticed the damage and failed to report it to the landlord', 'the tenant, because he is responsible for any damage that occurs during his lease period', 'the landlord, because of the contractual obligation under the lease to make all necessary repairs during the term of the lease.', 'the tenant, because he was negligent in maintaining the property', \\\"the landlord, because he didn't provide sufficient maintenance during the lease period\\\"]\",\"question\":\"A landlord was the owner of a two-story dwelling house and leased it completely furnished to a tenant for a 10-year period. Toward the end of the seventh year of the term, a violent storm blew off several of the roof shingles. The tenant noticed a leak through the roof where the shingles had blown off, but didn't make any repairs. A month later, a severe rain storm occurred and water leaked through the roof, causing damage to the valuable parquet floors in the two rooms below. In a subsequent proceeding by the landlord against the tenant to recover for the damage to the parquet floors resulting from the leak, liability would most likely be imposed upon\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}