{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"f1bc81a6-faf6-5ed5-9dcf-a3aa7b5b28c7","task_key":"default--test--1724","task_revision_id":"1","upstream_id":"1724","short_description":"In order to raise revenue, a city required home- repair contractors who…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['The court will find that the contract does violate the law and will not allow the contractor to recover his costs.', 'Although the contract violates the law, the court will require the homeowner to pay the contractor the full amount agreed upon.', \\\"The court will find that the contract is void due to the contractor's ignorance of the law.\\\", 'Although the contract violates the law, the court will find that public policy does not bar enforcement of the contract, because the purpose of the fee is merely to raise revenue.', 'Because the contractor was unaware of the fee, the court will rule in his favor and require the homeowner to pay for the repairs.', 'Because the purpose of the fee is merely to raise revenue, the court will find that the contract does not violate the law but will allow the contractor to recover his costs only.', 'The court will rule that the contractor must pay the licensing fee before any payment for the repairs can be made.', 'Because the contract violates the law and is void, the court will not enforce it.', 'The court will rule in favor of the homeowner, because the contractor failed to adhere to city regulations.']\",\"question\":\"In order to raise revenue, a city required home- repair contractors who performed work within the city limits to pay a licensing fee to a city agency. A contractor who was unaware of the fee requirement agreed to perform home repairs for a city resident. After the contractor completed the work, the resident discovered that the contractor had not paid the licensing fee, and she refused to pay for the repairs, which were otherwise satisfactory. If the contractor sues the resident for breach of contract, how is the court likely to rule?\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}