{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"0c2737bb-363e-552f-87b0-58a5f69a132b","task_key":"default--test--6089","task_revision_id":"1","upstream_id":"6089","short_description":"Which of the following is true about the carpal tunnel syndrome?","config":"default","split":"test","body":"{\"category\":\"health\",\"options\":\"['It causes numbness in the entire arm', 'It can be caused by rheumatoid arthritis', 'It can be cured by applying ice to the wrist', 'It is caused by compression of the radial nerve', 'It is a degenerative condition with no known cure', 'It is caused exclusively by overuse of the wrist', 'It can only be treated with surgery', 'Paraesthesia is worse during the day', 'It causes pain and tingling in the medial three and half digits', 'It is caused by compression of the ulna nerve']\",\"question\":\"Which of the following is true about the carpal tunnel syndrome?\",\"src\":\"ori_mmlu-clinical_knowledge\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}