{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"9f43d02a-c6e9-5843-848b-952cf56cd080","task_key":"default--test--1656","task_revision_id":"1","upstream_id":"1656","short_description":"Loretta Landlord owned a commercial building in a commercial strip surrounded by…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['Loretta, only if she can prove that Sally knew about the upcoming law change.', 'Loretta, if she can prove that Sally intended to breach the contract from the start.', 'Sally, because the change in law was beyond her control.', 'Sally, because the law change is a valid reason for terminating the lease.', 'Sally, because the law made performance of the lease contract impossible.', 'Sally, because the lease contract was discharged under impossibility of performance.', 'Loretta, only if she did not know of the use Sally intended for the space.', 'Loretta, because the new law was passed after the lease was executed.', 'Sally, because the law change made the purpose of the lease void.']\",\"question\":\"Loretta Landlord owned a commercial building in a commercial strip surrounded by a residential community. One of her largest spaces was rented on a month-to-month basis at $500 per month by a dance troop which held performances in the facility. The director of the dance troop moved to New York for a starring role Off-Broadway. Unfortunately the troop was unable to find another dance director, so it folded and vacated the space. Sally Sleazy learned of the closure and decided the location would be perfect for a topless strip joint. Sally rented the space from Loretta under a two-year lease at $1,000 per month without disclosing her intended use of the space. One month after the strip joint opened, the County Commissioners passed a new law prohibiting topless strip joints in that neighborhood. Sally told Loretta the new law made it impossible to operate the business, so she was leaving. If Loretta then sued Sally for breach of the lease contract, the likely outcome is a judgment for\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}