{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"98ca1a8f-70bb-545b-94ff-d9da7adeb6d5","task_key":"default--test--8444","task_revision_id":"1","upstream_id":"8444","short_description":"A town has one high school, which buses students from urban, suburban, and rural…","config":"default","split":"test","body":"{\"category\":\"math\",\"options\":\"['Stratified sample', 'Snowball sample', 'Simple random sample (SRS)', 'Voluntary response sample', 'Convenience sample', 'Cluster sample', 'Systematic sample', 'Purposive sample', 'Multistage sample', 'Quota sample']\",\"question\":\"A town has one high school, which buses students from urban, suburban, and rural communities. Which of the following samples is recommended in studying attitudes toward tracking of students in honors, regular, and below-grade classes?\",\"src\":\"ori_mmlu-high_school_statistics\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}