{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"0b97e0c2-1d9d-5508-9b5d-ba0b05922e26","task_key":"default--test--5808","task_revision_id":"1","upstream_id":"5808","short_description":"Heterosexual fantasies about sexual activity never involve someone _________,…","config":"default","split":"test","body":"{\"category\":\"other\",\"options\":\"['of the other gender; of the same gender', 'of the opposite gender; of the opposite gender', 'of the same gender; of the same gender', 'of the same gender; of the other gender', 'of the other gender; of the other gender', 'of the same gender; of the opposite gender', 'Both heterosexual and homosexual fantasies may involve persons of the same or other gender', 'Both heterosexual and homosexual fantasies may involve persons of the same gender only', 'Both heterosexual and homosexual fantasies may involve persons of the opposite gender only', 'of the opposite gender; of the same gender']\",\"question\":\"Heterosexual fantasies about sexual activity never involve someone _________, and gay and lesbian fantasies never involve persons of __________\",\"src\":\"ori_mmlu-human_sexuality\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}