{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"6f28b847-b096-53a6-92e2-7bd7109518e9","task_key":"default--test--11087","task_revision_id":"1","upstream_id":"11087","short_description":"Which of the following is a method of torture that Dershowitz believes is…","config":"default","split":"test","body":"{\"category\":\"philosophy\",\"options\":\"['prolonged sleep deprivation', 'forced ingestion of harmful substances', 'submerging the victim in water so he or she feels as if he or she is drowning', 'sensory deprivation', 'physical beatings', 'none of the above; Dershowitz thinks torture is wrong in all cases.', 'use of psychological manipulation techniques', \\\"having a sterilized needle shoved under the victim's fingernails\\\", 'exposure to extreme cold or heat', 'electrocution at nonlethal levels']\",\"question\":\" Which of the following is a method of torture that Dershowitz believes is appropriate in those cases where torture is permissible?\",\"src\":\"ori_mmlu-moral_disputes\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}