{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"13b00229-0cd3-59c7-9737-722cbaca51c2","task_key":"all--test--13b00229-0cd3-59c7-9737-722cbaca51c2","task_revision_id":"1","upstream_id":"","short_description":"According to usual practice, a long-time regular customer left her car at an…","config":"all","split":"test","body":"{\"choices\":\"['No, because there is always an implication that the provider must get permission before doing work not approved.', 'Yes, because the circumstances and past practice established an implied-in-law contract based on mutual assent.', 'No, because a contract cannot be based on implications and guess work.', 'Yes, because the circumstances, including prior practice, showed intent to contract and a meeting of the minds.']\",\"question\":\"According to usual practice, a long-time regular customer left her car at an auto mechanic's shop with written instructions to \\\"check and tune up the engine; make sure the brakes are good and working; examine the tires; also, check and repair all major systems because I am driving cross country next week.\\\" She did not see the mechanic or discuss with him the price or other details. When she picked up the car, there was a bill for $5,000 on the dash board. All of the services and parts were workmanlike, not defective and pertained to the areas in the instructions. She refused to pay, claiming that there was no agreement and that the mechanic should have called for advance approval. Can she be held responsible for contractual services performed?\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}