{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"cebd8311-893e-5c42-a6d6-cb7275eae788","task_key":"all--test--cebd8311-893e-5c42-a6d6-cb7275eae788","task_revision_id":"1","upstream_id":"","short_description":"A well-known Washington newspaper columnist was researching the background of a…","config":"all","split":"test","body":"{\"choices\":\"[\\\"prevail, because the apprentice's action was an unlawful invasion of private facts.\\\", 'prevail, because the publication was not newsworthy.', 'not prevail, because what the columnist printed was true, thus establishing there was no \\\"false light. \\\"', 'not prevail, because the columnist was not involved in the burglary and did not conspire with the apprentice with respect to the burglary.']\",\"question\":\"A well-known Washington newspaper columnist was researching the background of a lawyer who had been nominated to become attorney general of the United States and was awaiting confirmation by the U. S. Senate. One of the columnist's eager young apprentices concocted a plan that he hoped would enable him to gain the columnist's favor. Late one evening, without telling anyone, the apprentice broke into the lawyer's private office and copied several letters pertaining to \\\"dirty tricks\\\" perpetrated by the lawyer during the most recent presidential campaign. When the apprentice presented the columnist with the fruits of his diligent research, the columnist was quite impressed and immediately printed excerpts from the material. If the lawyer asserts a claim against the columnist for invasion of right to privacy, the plaintiff will most likely\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}