{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"b9266778-43c6-5ff8-a3ba-c0105a5a7371","task_key":"default--test--6425","task_revision_id":"1","upstream_id":"6425","short_description":"You suspect that your patient has an enlarged submandibular salivary gland. You…","config":"default","split":"test","body":"{\"category\":\"health\",\"options\":\"['to be palpable neither intra- nor extraorally.', 'to be palpable both intra- and extraorally.', 'only to be detectable by palpating the adjacent tissues.', 'only to be detectable by a CT scan.', 'to be palpable intraorally and detectable by radiographical examination.', 'to be palpable intraorally..', 'to be palpable extraorally.', 'to be palpable extraorally and detectable by ultrasound examination.', 'only to be detectable by radiographical examination.', 'only to be detectable by ultrasound examination.']\",\"question\":\"You suspect that your patient has an enlarged submandibular salivary gland. You expect the enlarged gland\",\"src\":\"ori_mmlu-anatomy\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}