{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"82194c01-2e38-5d88-88de-c82765fe929f","task_key":"all--test--82194c01-2e38-5d88-88de-c82765fe929f","task_revision_id":"1","upstream_id":"","short_description":"A woman needed to have her microwave repaired. She contacted the local handyman,…","config":"all","split":"test","body":"{\"choices\":\"['The woman has a cause of action against the repair store only.', 'The woman has a cause of action against the handyman only.', 'The woman has a cause of action against both the repair store and the handyman.', 'The woman has no cause of action against either the repair store or the handyman.']\",\"question\":\"A woman needed to have her microwave repaired. She contacted the local handyman, who said he could repair the microwave for $100. The woman readily agreed and delivered the microwave to the handyman, who promised to have it ready in two weeks. One week later, the handyman realized that he had so much work to do that he would not be able to repair the microwave on time. He then took the microwave to a repair store. The repair store agreed to repair the microwave for $80 within one week. The owner of the repair store that contracted with the handyman was unaware that the woman actually owned the microwave. If the repair store fails to repair the microwave, which of the following is the most accurate statement?\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}