{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"7edfaed0-f790-5902-a81d-624761d02153","task_key":"default--test--1797","task_revision_id":"1","upstream_id":"1797","short_description":"A homeowner executed a deed by which he conveyed his home and surround property…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['vested remainder subject to open.', 'contingent remainder.', 'fee simple subject to executory limitation.', 'reversionary interest.', 'life estate pur autre vie.', 'shifting executory interest.', 'fee simple determinable.', 'fee simple subject to condition subsequent.', 'vested remainder subject to partial divestiture.', 'vested remainder subject to complete divestiture.']\",\"question\":\"A homeowner executed a deed by which he conveyed his home and surround property for a consideration of one dollar, receipt of which was acknowledged, \\\"to my daughter for life, then to my aunt for life, but if my aunt moves to another state, to my sister for the life of my aunt, then to the heirs of my aunt if my aunt does not move to another state, and to the heirs of my sister if my aunt does move to another state. \\\" This deed was promptly recorded. During the daughter's lifetime, the aunt's interest may best be described as a\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}