{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"269682f4-afa1-5618-8b10-6fc4499789c5","task_key":"default--test--9098","task_revision_id":"1","upstream_id":"9098","short_description":"A boat travels directly upstream in a river, moving with constant but unknown…","config":"default","split":"test","body":"{\"category\":\"physics\",\"options\":\"['2 miles per hour', '1 mile per hour', '2.5 miles per hour', '3 miles per hour', '5 miles per hour', '4 miles per hour', '1.5 miles per hour', '0.5 miles per hour', '0.75 miles per hour', '3.5 miles per hour']\",\"question\":\"A boat travels directly upstream in a river, moving with constant but unknown speed v with respect to the water. At the start of this trip upstream, a bottle is dropped over the side. After 15 minutes the boat turns around and heads downstream. It catches up with the bottle when the bottle has drifted one mile downstream from the point at which it was dropped into the water. What is the current in the stream?\",\"src\":\"stemez-Physics\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}