{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"faf432a0-a449-5aed-aa04-fffd1898fa6d","task_key":"default--test--1340","task_revision_id":"1","upstream_id":"1340","short_description":"A deed executed by a woman in 2001 conveyed a tract of land for a consideration…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['The woman.', 'The cousin and the woman equally.', \\\"The cousin's heirs.\\\", 'The cousin.', 'The state.', \\\"The woman's heirs.\\\", \\\"Divided equally between the friend's heirs and the woman.\\\", \\\"The friend's next of kin.\\\", \\\"The friend's heirs.\\\"]\",\"question\":\"A deed executed by a woman in 2001 conveyed a tract of land for a consideration of one dollar, receipt of which was acknowledged, \\\"to my friend for life, but if liquor is ever sold on the tract of land, then to my cousin and his heirs, and if for any reason the interest hereby conveyed to my cousin is not valid, then I reserve the right to re-enter the tract of land and take back my property. \\\" In 2004, the friend died intestate before the wheat he had planted could be harvested. Who is entitled to the proceeds of the crop?\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}