{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"b56356c6-3800-563a-9e58-3e408b5c727c","task_key":"default--test--2530","task_revision_id":"1","upstream_id":"2530","short_description":"Which of the following is the best example of an attribute that is…","config":"default","split":"test","body":"{\"category\":\"psychology\",\"options\":\"['Having the desire to reproduce', 'Following a particular dress code', 'Arriving on time for work', 'Fearing death', 'Preference for certain types of foods', 'Responding to physical pain', \\\"Caring for one's children\\\", 'Needing sleep', 'Seeking safety', 'Seeking food and water']\",\"question\":\"Which of the following is the best example of an attribute that is culturally-based rather than primarily psychologically-based?\",\"src\":\"ori_mmlu-high_school_psychology\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}