{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"35820db5-3eb8-5289-8e73-bdb7e9f55860","task_key":"default--test--6180","task_revision_id":"1","upstream_id":"6180","short_description":"A 43-year-old female presents to the office with a 2-month history of heat…","config":"default","split":"test","body":"{\"category\":\"health\",\"options\":\"['diffuse enlargement of the rectus muscles', 'cerebral aneurysm', 'pituitary adenoma', 'normal brain tissue', 'inflammation of the optic nerve', 'calcification of the orbit', 'atrophy of the optic nerve', 'dilated ventricles', 'lymphoma', 'brain metastases']\",\"question\":\"A 43-year-old female presents to the office with a 2-month history of heat intolerance, weight loss, tachycardia, and diffuse goiter. Examination reveals exophthalmos, inflammation of the periorbital tissue and conjunctivae, a heart rate of 140/min, and warm, velvety skin. An MRI of the head and orbits will most likely reveal \",\"src\":\"ori_mmlu-professional_medicine\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}