{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"f37804c7-d6a7-5214-af1e-469d330837e9","task_key":"default--test--2778","task_revision_id":"1","upstream_id":"2778","short_description":"Today Susan took a pill for her allergy that raised her blood pressure, caused…","config":"default","split":"test","body":"{\"category\":\"psychology\",\"options\":\"['two-factor theory', 'Incentive theory', 'Opponent-process theory', 'the adaptation-level phenomenon', 'Cognitive dissonance theory', 'Arousal theory', 'Cannon-Bard theory', 'Cognitive Appraisal theory', 'James-Lange theory', 'Drive-reduction theory']\",\"question\":\"Today Susan took a pill for her allergy that raised her blood pressure, caused her heart to beat faster, and raised her body temperature. Now caught in traffic, she feels angry. Yesterday, when she took the pill, she was with her husband. When her blood pressure rose, her heart speeded up, she got hotter, and she felt amorous. This description exemplifies\",\"src\":\"ori_mmlu-high_school_psychology\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}