{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"490cab34-110b-59fb-981b-fbf627b17488","task_key":"default--test--6350","task_revision_id":"1","upstream_id":"6350","short_description":"A patient suffers a broken neck with damage to the spinal cord at the level of…","config":"default","split":"test","body":"{\"category\":\"health\",\"options\":\"['It is impossible to predict an effect on breathing.', 'Their breathing rate will significantly increase.', 'They will only be able to breathe quietly.', 'Their breathing will be erratic and unpredictable.', 'They will be unable to breathe without life support.', 'Breathing will be unaffected.', 'They will be able to breathe only with the help of a ventilator.', 'They will be able to breathe only in short, shallow breaths.', 'They will have to breathe through a tracheostomy tube.', 'They will not be able to breathe out, only in.']\",\"question\":\"A patient suffers a broken neck with damage to the spinal cord at the level of the sixth cervical vertebra.\",\"src\":\"ori_mmlu-anatomy\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}