{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"8f449837-4247-5e29-8fed-128bb3e326a7","task_key":"default--test--6179","task_revision_id":"1","upstream_id":"6179","short_description":"A 43-year-old female presents to the office with a 2-month history of heat…","config":"default","split":"test","body":"{\"category\":\"health\",\"options\":\"['lymphoma', 'atrophy of the optic nerve', 'normal brain tissue', 'calcification of the orbit', 'diffuse enlargement of the rectus muscles', 'pituitary adenoma', 'dilated ventricles', 'cerebral aneurysm', 'inflammation of the optic nerve', 'brain metastases']\",\"question\":\"A 43-year-old female presents to the office with a 2-month history of heat intolerance, weight loss, tachycardia, and diffuse goiter. Examination reveals exophthalmos, inflammation of the periorbital tissue and conjunctivae, a heart rate of 140/min, and warm, velvety skin. An MRI of the head and orbits will most likely reveal\",\"src\":\"ori_mmlu-professional_medicine\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}