{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"f83c093d-b3c1-5217-87ae-ccd7c8bdc5a0","task_key":"all--test--f83c093d-b3c1-5217-87ae-ccd7c8bdc5a0","task_revision_id":"1","upstream_id":"","short_description":"A man needed to have the oil changed on his car. On Friday, he decided to take…","config":"all","split":"test","body":"{\"choices\":\"['win, because she performed her part of the bargain.', 'win, because the second contract for $75 superseded the original $50 contract.', 'lose, because the $75 contract did not supersede the $50 contract.', 'lose, because the neighbor had a pre-existing duty to change the oil in the car for $50.']\",\"question\":\"A man needed to have the oil changed on his car. On Friday, he decided to take his car to the local dealership to have the oil changed and asked his neighbor if she would give him a ride home from the dealership. The neighbor said, \\\"Why pay the high prices a dealership will charge you? I can change the oil in your car for you. If you will agree to pay me $50, I'll change the oil in your car over the weekend. \\\" The man readily agreed. On Sunday afternoon, the man noticed that his neighbor still had not started working on the car. He asked his neighbor if the car would be ready for him to drive to work Monday morning. The neighbor replied, \\\"I thought about it and realized $50 is too low a price for the work involved. I don't think I'm going to change the oil in your car. \\\" The man then said, \\\"Look, I realize $50 is low for the work involved. If you can change the oil in my car by tomorrow morning, I'll pay you an additional $25. And I won't sue you in small claims court for your failure to perform your promise. \\\" The neighbor then changed the oil late Sunday afternoon, but the man refused to pay to the neighbor anything more than $50. In a suit by the neighbor to recover the additional $25 promised by the man, the neighbor will\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}