{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"5318f147-f521-56e7-ae2c-e271c5eefcf0","task_key":"all--test--5318f147-f521-56e7-ae2c-e271c5eefcf0","task_revision_id":"1","upstream_id":"","short_description":"A woman was employed as a legal secretary for a local attorney. After the…","config":"all","split":"test","body":"{\"choices\":\"[\\\"prevail, because the first attorney's statement reflected adversely on the woman's professional competence.\\\", \\\"prevail, because the first attorney was mistaken in the facts upon which he based his opinion of the woman's performance.\\\", 'not prevail, because the first attorney had reasonable grounds for his belief that the woman was incompetent.', 'not prevail, because the woman listed her former employment with the first attorney in her résumé to the second attorney.']\",\"question\":\"A woman was employed as a legal secretary for a local attorney. After the attorney terminated the woman's employment, she sent a job application and résumé to another attorney. In the résumé she sent to the second attorney, the woman listed her former employment with the first attorney. After receiving the woman's résumé, the second attorney telephoned the first attorney for his opinion of her qualifications. The first attorney replied that he dismissed the woman \\\"because she was unprofessional and incompetent. \\\" The first attorney's assessment was based on one malpractice incident for which he blamed the woman, but which, in fact, was attributable to another secretary in his law firm. Although the first attorney reasonably believed that his low rating of the woman was a fair reflection of her performance, he wrongfully held her responsible for the malpractice incident. Based on the first attorney's poor recommendation, the second attorney did not hire the woman. In a defamation action by the woman against the first attorney, the plaintiff will most likely\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}