{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"31310938-7db8-513d-9a2e-76384a6384a9","task_key":"default--test--1186","task_revision_id":"1","upstream_id":"1186","short_description":"A widget manufacturer and a widget retailer signed the following agreement:\"The…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['a bilateral contract.', 'a periodic contract.', '10 separate contracts.', 'a contract of adhesion.', 'a conditional contract.', 'a requirements-output contract.', 'a divisible contract.', 'a unilateral contract.', 'a void contract.', 'an installment contract.']\",\"question\":\"A widget manufacturer and a widget retailer signed the following agreement:\\\"The manufacturer promises to sell and the retailer promises to buy 10,000 widgets at $50 per 100. One thousand widgets are to be delivered by the manufacturer on the first day of each of the next 10 months. EACH MONTHLY DELIVERY ISA SEPARATE CONTRACT. Payment to be made within 10 days of receipt of each shipment. \\\"The agreement between the manufacturer and the retailer may best be interpreted as\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}