{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"8b93425d-2144-5e94-aecf-0efb59650e2e","task_key":"default--test--1076","task_revision_id":"1","upstream_id":"1076","short_description":"A grantor executed a deed by which he conveyed his apartment building for a…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['contingent remainder pur autre vie.', 'vested remainder absolute.', 'estate pur autre vie.', 'shifting executory interest pur autre vie.', 'vested remainder pur autre vie.', 'contingent remainder in fee simple.', 'contingent executory interest pur autre vie.', 'vested remainder in fee simple.', 'vested executory interest pur autre vie.', 'contingent remainder absolute.']\",\"question\":\"A grantor executed a deed by which he conveyed his apartment building for a consideration of one dollar, receipt of which was acknowledged, \\\"to my son for life, then to my uncle for life, but if my uncle moves to another state, to my brother for the life of my uncle, then to the heirs of my uncle if my uncle does not move to another state, and to the heirs of my brother if my uncle does move to another state. \\\" This deed was promptly recorded. During the son's lifetime, the brother's interest may best be described as a (an)\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}