{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"057d6965-2cbc-53ad-bc8c-25171ddcc100","task_key":"default--test--1779","task_revision_id":"1","upstream_id":"1779","short_description":"An American franchise operates as an importer and distributor of bicycles…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['tires, but only while they are in the warehouse.', 'bicycles, but only if they are not in their shipping crates.', 'neither the tires nor the bicycles, unless they are being actively distributed.', 'bicycles only.', 'neither the tires nor the bicycles.', 'bikes and tires, but only during the distribution process.', 'bicycles, but only while they are on the loading docks.', 'tires only.', 'bikes and tires, but only when they are stored in the warehouse.', 'tires and bicycles.']\",\"question\":\"An American franchise operates as an importer and distributor of bicycles manufactured by a foreign country, and maintains several warehouses for the wholesale distribution of the bicycles. A warehouse located in a county handles the distribution of bicycles for several states. The bikes and bike tires are shipped separately to the county warehouse. The tires are mingled, stacked, and stored in the warehouse along with various other tires. The bicycles, on the other hand, arrive completely assembled in their shipping crates and remain on the loading docks. The county imposes an ad valorem property tax on the bikes and tires. The county's ad valorem tax may properly be assessed against the \",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}