{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"4b5b074a-8027-5661-b42d-26757a8cbb99","task_key":"default--test--10291","task_revision_id":"1","upstream_id":"10291","short_description":"A train is traveling at a constant speed of 60 mph and is picking up water from…","config":"default","split":"test","body":"{\"category\":\"physics\",\"options\":\"['2750lbw and 500 hp', '3000lbw and 475 hp', '2900lbw and 465 hp', '3200lbw and 510 hp', '3100lbw and 495 hp', '3025lbw and 484 hp', '3500lbw and 550 hp', '2800lbw and 450 hp', '2500lbw and 400 hp', '2600lbw and 415 hp']\",\"question\":\"A train is traveling at a constant speed of 60 mph and is picking up water from a trough between the rails. If it picks up 15,000 lb of water in 400 yd, determine the exact force and horsepower required.\",\"src\":\"stemez-Mechanics\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}