{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"c7fe3e40-ddf2-5986-9f25-f3b38640cf04","task_key":"default--test--6419","task_revision_id":"1","upstream_id":"6419","short_description":"A patient is stabbed in the neck. You suspect damage to the accessory nerve in…","config":"default","split":"test","body":"{\"category\":\"health\",\"options\":\"['turn their head to the side without impairment.', 'lift their shoulders against resistance.', 'extend their neck against resistance.', 'extend their neck without impairment.', 'rotate their head against resistance.', 'shrug their shoulders without resistance.', 'lift their shoulders without impairment.', 'rotate their head without impairment.', 'turn their head to the side against resistance.', 'extend their arm against resistance.']\",\"question\":\"A patient is stabbed in the neck. You suspect damage to the accessory nerve in the posterior triangle. You would test nerve function by asking the patient to\",\"src\":\"ori_mmlu-anatomy\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}