{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"b2882ff4-94a5-5386-b829-1a4043002a93","task_key":"default--test--8685","task_revision_id":"1","upstream_id":"8685","short_description":"There are 2,817 homes in the town of West Valley. Each home uses an average of…","config":"default","split":"test","body":"{\"category\":\"math\",\"options\":\"['860,460 gallons', '1,060,460 gallons', '1,150,460 gallons', '1,070,460 gallons', '870,460 gallons', '900,460 gallons', '980,460 gallons', '1,080,460 gallons', '1,090,460 gallons', '1,100,460 gallons']\",\"question\":\"There are 2,817 homes in the town of West Valley. Each home uses an average of 380 gallons of water each day. Use the expression below to find the total number of gallons of water the homes in West Valley use on average each day. 2,817 × 380 What is the total number of gallons of water the homes in West Valley use on average each day?\",\"src\":\"ori_mmlu-elementary_mathematics\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}