{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"45c82d95-3629-5e7d-a3c4-f100f15c1f02","task_key":"default--test--9820","task_revision_id":"1","upstream_id":"9820","short_description":"A sphere, a cube, and a cylinder, all of equal mass, are released from rest from…","config":"default","split":"test","body":"{\"category\":\"physics\",\"options\":\"['The object that is released first', 'The object that is released last', 'The cube and the cylinder reach the base at the same time, before the sphere', 'the sphere', 'the cylinder', 'the cube', 'The object in the middle', 'All reach the base at the same time.', 'None of them reach the base', 'The sphere and the cylinder reach the base at the same time, before the cube']\",\"question\":\"A sphere, a cube, and a cylinder, all of equal mass, are released from rest from the top of a short incline. The surface of the incline is extremely slick, so much so that the objects do not rotate when released, but rather slide with negligible friction. Which reaches the base of the incline first?\",\"src\":\"ori_mmlu-high_school_physics\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}