{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"a89829cc-5b82-5c38-ac6b-c765b653da06","task_key":"default--test--1702","task_revision_id":"1","upstream_id":"1702","short_description":"Albert Attorney was a solo practitioner in Littletown who operated a general law…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['Albert did not intentionally make the mistake in the estate plan.', \\\"The statute of limitations has run, so Gordon's lawsuit is not timely.\\\", 'Gordon was not an eligible taker under the estate plan.', 'The error in the estate plan was due to a change in legislation.', 'The rule against perpetuities is too complicated for the average lawyer in the locality of Littletown to understand.', \\\"Albert Attorney was not practicing at the time of the testator's death.\\\", \\\"Albert was not aware of the mistake until the testator's death.\\\", 'The legal fee for the original estate plan was never paid.', 'Gordon Greedy was not a direct client of Albert Attorney.', 'Gordon Greedy was not a relative and therefore cannot sue for professional malpractice.']\",\"question\":\"Albert Attorney was a solo practitioner in Littletown who operated a general law practice. In 1990, Albert drafted a document for a client's estate plan that unfortunately violated the rule against perpetuities. Upon the testator's death in 2004, the mistake was discovered. Because of the error, the estate plan was not admitted to probate so the assets were distributed under the state intestacy statute. Gordon Greedy was named as a substantial taker under the estate plan, but did not take under the intestate statute since he was not a relative. The state statute of limitations for professional malpractice is three years. If Gordon sues Albert for professional malpractice in 2005, the best defense Albert can assert is\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}