{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"5f5b4acd-e6e3-5aa6-b4c7-ce77919dc30b","task_key":"default--test--5377","task_revision_id":"1","upstream_id":"5377","short_description":"Which of the following is typically associated with having a vasectomy?","config":"default","split":"test","body":"{\"category\":\"other\",\"options\":\"['decreased sperm production', 'inability to maintain an erection', 'increased risk of prostate cancer', 'decreased testosterone production', 'increased risk of heart disease', 'normal orgasm', 'reduced rigidity of erections', 'infertility due to absence of sperm in semen', 'inability to produce semen', 'loss of sexual desire']\",\"question\":\"Which of the following is typically associated with having a vasectomy?\",\"src\":\"ori_mmlu-human_sexuality\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}