{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"f4cf36c8-04ed-5975-a0c3-2658b15b694f","task_key":"default--test--1793","task_revision_id":"1","upstream_id":"1793","short_description":"A resident of a state owned a large tract of virgin forest. The resident…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['easement in gross.', 'Fee simple.', 'easement appurtenant.', 'Joint tenancy.', 'Leasehold estate.', 'Tenancy in common.', 'profit-a-prendre.', 'Life estate.', 'Quitclaim deed.', 'license.']\",\"question\":\"A resident of a state owned a large tract of virgin forest. The resident conveyed this tract \\\"to the people of the state in fee simple so long as said tract shall remain in its natural state (with exception for recreational, scientific, or educational purposes). \\\"Under state law, conveyances of real property to \\\"the people\\\" of the state are deemed to vest title in the state regarded as a legal entity, and custody over such property resides in an appointed official known as the director of environmental resources. The director granted a logging company the privilege to cut timber on a remote portion of the tract, together with incidental easements of way. The right of the logging company to cut and sell the timber that was to be cleared from the tract would be an example of a (an)\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}