{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"200f8efa-b1af-5d2e-a19c-5ecd296885aa","task_key":"default--test--1778","task_revision_id":"1","upstream_id":"1778","short_description":"An American franchise operates as an importer and distributor of bicycles…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['bikes and tires, but only when they are stored in the warehouse.', 'bicycles only.', 'tires, but only while they are in the warehouse.', 'tires and bicycles.', 'bikes and tires, but only during the distribution process.', 'bicycles, but only while they are on the loading docks.', 'bicycles, but only if they are not in their shipping crates.', 'neither the tires nor the bicycles, unless they are being actively distributed.', 'neither the tires nor the bicycles.', 'tires only.']\",\"question\":\"An American franchise operates as an importer and distributor of bicycles manufactured by a foreign country, and maintains several warehouses for the wholesale distribution of the bicycles. A warehouse located in a county handles the distribution of bicycles for several states. The bikes and bike tires are shipped separately to the county warehouse. The tires are mingled, stacked, and stored in the warehouse along with various other tires. The bicycles, on the other hand, arrive completely assembled in their shipping crates and remain on the loading docks. The county imposes an ad valorem property tax on the bikes and tires. The county's ad valorem tax may properly be assessed against the\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}