{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"0b4d7ae0-ad16-5779-8353-f46f9980e584","task_key":"all--test--0b4d7ae0-ad16-5779-8353-f46f9980e584","task_revision_id":"1","upstream_id":"","short_description":"A purchaser filed a federal diversity action against a seller, alleging breach…","config":"all","split":"test","body":"{\"choices\":\"['File a reply that includes a denial of the separate defense.', 'Move for sanctions against the seller for asserting a frivolous defense.', 'Move to amend the complaint to add allegations about the differences between the lawsuits.', 'Move to strike the separate defense as irrelevant.']\",\"question\":\"A purchaser filed a federal diversity action against a seller, alleging breach of contract. The seller answered the complaint and included as a separate defense an allegation that the purchaser had brought and lost a similar contract claim against a different seller three years earlier, and that this history represented a pattern of filing frivolous lawsuits. The purchaser believes that the earlier lawsuit was factually completely different from the current one and is therefore irrelevant. What is the purchaser's best response to the seller's answer?\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}