{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"8b9ef3e9-5e65-58b0-bbfe-edb50121e9f9","task_key":"default--test--1555","task_revision_id":"1","upstream_id":"1555","short_description":"A well-known Washington newspaper columnist was researching the background of a…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['prevail, because the information was obtained illegally.', 'prevail, because the publication was not newsworthy.', \\\"prevail, because the apprentice's action was an unlawful invasion of private facts.\\\", 'prevail, because the apprentice acted on behalf of the columnist, making him liable.', 'not prevail, because the apprentice acted independently and without the knowledge or direction of the columnist.', 'not prevail, because the columnist was not involved in the burglary and did not conspire with the apprentice with respect to the burglary.', 'not prevail, because the information was of public interest and its publication served a greater public good.', 'not prevail, because the lawyer was a public figure and therefore his actions are subject to public scrutiny.', 'prevail, because the publication of private letters is an infringement on personal rights.', 'not prevail, because what the columnist printed was true, thus establishing there was no \\\"false light. \\\"']\",\"question\":\"A well-known Washington newspaper columnist was researching the background of a lawyer who had been nominated to become attorney general of the United States and was awaiting confirmation by the U. S. Senate. One of the columnist's eager young apprentices concocted a plan that he hoped would enable him to gain the columnist's favor. Late one evening, without telling anyone, the apprentice broke into the lawyer's private office and copied several letters pertaining to \\\"dirty tricks\\\" perpetrated by the lawyer during the most recent presidential campaign. When the apprentice presented the columnist with the fruits of his diligent research, the columnist was quite impressed and immediately printed excerpts from the material. If the lawyer asserts a claim against the columnist for invasion of right to privacy, the plaintiff will most likely\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}