{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"be03c548-4449-5ac3-a7c5-56a0c5a5e4b2","task_key":"default--test--1611","task_revision_id":"1","upstream_id":"1611","short_description":"A condominium development consists of two buildings, one with balconies attached…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['Yes, because families with children are already living in units with balconies.', 'No, because the amendment was promptly recorded and is legally binding.', 'Yes, because families with children cannot be segregated within the condominium development.', 'No, because the association is acting in the best interests of child safety.', 'No, because families with children are allowed to purchase units in the building without balconies.', 'No, because the association has the right to approve all sales.', 'No, because the amendment is based on legitimate safety issues.']\",\"question\":\"A condominium development consists of two buildings, one with balconies attached to each unit, and one withno balconies. For safety concerns, the condominium association amended the covenants and restrictions toprohibit future sales of balcony units to families with minor children. The amendment did not affect families withchildren already living in balcony units. The amendment was promptly recorded. The condominium associationhad a valid covenant providing that all sales had to be approved by the association. Subsequent to the effective date of the amendment, the owner of a balcony unit contracted to sell it to a familywith minor children. Before the closing, the association told the buyers that because they had minor children,they could not buy the unit. The association further told the buyers that numerous units were available in thebuilding without balconies. After receiving this notification, the buyers complained to a fair housing agency, claiming that the amendmentwas unenforceable because it violated federal fair housing laws. Is there reasonable cause to believe that a violation has occurred?\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}