{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"b4755981-6960-54c8-b8e8-a85ac7470843","task_key":"all--test--b4755981-6960-54c8-b8e8-a85ac7470843","task_revision_id":"1","upstream_id":"","short_description":"An owner took his car to an auto mechanic to have the oil changed. When he…","config":"all","split":"test","body":"{\"choices\":\"['not guilty, because the car was his own property.', 'not guilty, if the jury finds the servicing charge to be excessively high.', 'guilty, if the jurisdiction has a statute making theft of services a crime.', \\\"guilty, because he took the car without the mechanic's consent.\\\"]\",\"question\":\"An owner took his car to an auto mechanic to have the oil changed. When he returned to pick up his car later in the day, the mechanic told him the charge was $1 The owner objected and said the charge was excessively high. The mechanic indicated that the amount was reasonable and showed the owner a price listing substantiating the charge. As the two men were arguing, the mechanic excused himself to answer the telephone in the rear of his shop. While the mechanic was on the phone, the owner got into his car and drove off without paying the servicing charges. If the owner is subsequently charged with larceny, he should be found\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}