{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"7b6d73c4-ff8f-5763-ab2b-627fdd11fe80","task_key":"all--test--7b6d73c4-ff8f-5763-ab2b-627fdd11fe80","task_revision_id":"1","upstream_id":"","short_description":"A contractor and home owner were bargaining on the price for the construction of…","config":"all","split":"test","body":"{\"choices\":\"['The home owner asks the contractor if they would be willing to build the house for $95,000.', 'The contractor contacts the home owner and states that the offer is withdrawn.', \\\"The contractor dies before the home owner accepts but the contractor's son intends to continue the business.\\\", 'The home owner states \\\"I accept your offer but the price is to be $97,000.\\\"']\",\"question\":\"A contractor and home owner were bargaining on the price for the construction of a new home. The contractor made a number of offers for construction to the home owner including one for $100,000. Which of the following communications would not terminate the offer so that a subsequent acceptance could be effective\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}