{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"4037fb8b-515b-555a-a1eb-006249f5baf4","task_key":"default--test--6228","task_revision_id":"1","upstream_id":"6228","short_description":"When an emergency opening into the airway is required, the cricothyroid membrane…","config":"default","split":"test","body":"{\"category\":\"health\",\"options\":\"['Superior to the hyoid bone.', 'At the level of the vocal cords.', 'inferior to the thyroid cartilage.', 'inferior to the cricoid cartilage.', 'superior to the thyroid isthmus.', 'superior to the thyroid cartilage.', 'At the level of the epiglottis.', 'At the level of the trachea.']\",\"question\":\"When an emergency opening into the airway is required, the cricothyroid membrane should be pierced immediately\",\"src\":\"ori_mmlu-anatomy\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}