{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"c5586bd6-2324-5583-8287-c63fba5be710","task_key":"default--test--4924","task_revision_id":"1","upstream_id":"4924","short_description":"This question refers to the following information.","config":"default","split":"test","body":"{\"category\":\"history\",\"options\":\"['A call for personal sacrifice in the name of faith', 'A sermon about respecting political authority', 'An invitation to question traditional religious beliefs', 'An ecumenical call for religious tolerance', 'A cautionary tale about the dangers of hedonism', 'A mystical appeal to the divine in nature', 'An endorsement of joyful celebration', 'A philosophical discourse on the nature of sin and redemption', 'A puritanical injunction to adhere to a strict code of conduct', 'A spirited attempt to convert non-believers']\",\"question\":\"This question refers to the following information.\\nO ye who believe! Strong drink and games of chance and idols… are only an infamy of Satan's handiwork. Leave it aside in order that ye may succeed.\\nthe Qur'an, Sura 5: 89\\nWhen once you hear the roses are in bloom,\\nThen is the time, my love, to pour the wine.\\nthe Rubáiyát of Omar Khayyám\\nThe tone of the first verse best reflects which of the following modes of religious discourse?\",\"src\":\"ori_mmlu-high_school_world_history\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}