{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"d3bad4de-7e45-5cdd-960e-87a12a696e26","task_key":"default--test--1549","task_revision_id":"1","upstream_id":"1549","short_description":"On March 1, a homeowner and a painter entered into a written contract wherein…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['No, because the homeowner is estopped to deny the validity of the written contract.', 'Yes, because the homeowner was not able to meet the financial obligations of the contract.', 'No, because the painter can claim damages for the breach of contract.', 'No, because the homeowner did not make efforts to secure a loan from another bank.', 'No, because the agreement regarding the loan varied the express terms of the writing.', 'Yes, because the loan agreement was a condition precedent to the existence of the contract.', 'No, because the oral agreement about the loan does not hold legal validity.', 'Yes, because the painter was informed about the condition of the loan before the contract was signed.', 'Yes, because the agreement regarding the loan constituted a valid modification of the writing.']\",\"question\":\"On March 1, a homeowner and a painter entered into a written contract wherein the painter promised to paint the exterior of the homeowner's house for $3,000. Prior to the signing of the contract, the homeowner and the painter orally agreed that their contract would be null and void unless the homeowner was able to obtain a $3,000 loan from National Bank before April 1. On March 31, the homeowner was informed by National Bank that his loan application had been rejected. The next day, the homeowner telephoned the painter and informed him that the deal was off. If the painter brings an action for breach of contract against the homeowner, would the latter's inability to secure the loan provide him with a valid defense?\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}