{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"d0bcfab1-0f90-5e2c-998f-f21b82e59491","task_key":"default--test--2374","task_revision_id":"1","upstream_id":"2374","short_description":"Pilocarpineis a drug that stimulates the nerve endings of parasympatheticnerves.…","config":"default","split":"test","body":"{\"category\":\"psychology\",\"options\":\"['(a) Digestion would be slightly enhanced, (b) the pupil size would not change, (c) the heart rate would decrease moderately.', '(a) No significant effect on digestion, (b) the pupil would constrict, (c) the heart rate would increase markedly.', '(a) Digestion would become highly irregular, (b) the pupil would constrict slightly, (c) the heart rate would increase dramatically.', '(a) Digestion would be slightly inhibited, (b) the pupil would dilate, (c) the heart rate would remain unchanged.', '(a) Digestion is facilitated or enhanced, (b) the pupil would reduce in size, (c) the heart beat would slow and weaken.', '(a) Digestion is inhibited, (b) the pupil would enlarge, (c) the heart beat would increase and strengthen.', '(a) Digestion would be unaffected, (b) the pupil would fluctuate in size, (c) the heart rate would decrease slightly.', '(a) Digestion would slow down, (b) the pupil would remain the same size, (c) the heart beat would quicken.', '(a) Digestion would speed up dramatically, (b) the pupil would dilate significantly, (c) the heart rate would increase slightly.', '(a) Digestion remains unaffected, (b) the pupil would dilate, (c) the heart beat would remain the same.']\",\"question\":\"Pilocarpineis a drug that stimulates the nerve endings of parasympatheticnerves. What effects would you expect this drugto have on (a) the digestive tract, (b) the iris of the eye, and(c) the heart rate?\",\"src\":\"stemez-Psychology\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}