{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"e19f9a44-c261-5c36-91e6-c35be34bb248","task_key":"default--test--4339","task_revision_id":"1","upstream_id":"4339","short_description":"Which is correct about the calcium atom?","config":"default","split":"test","body":"{\"category\":\"chemistry\",\"options\":\"['It contains 20 protons and neutrons', 'All atoms of calcium have 20 protons and 40 neutrons', 'It contains 20 protons, neutrons, and electrons', 'All calcium atoms have 20 protons, neutrons, and electrons', 'All atoms of calcium have a mass of 40.078 u', 'It contains 40 protons and 40 electrons', 'All calcium atoms have a mass of 20.078 u', 'It contains 20 protons and 20 electrons', 'It contains 20 neutrons and 20 electrons', 'It contains 40 protons and 20 electrons']\",\"question\":\"Which is correct about the calcium atom?\",\"src\":\"ori_mmlu-high_school_chemistry\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}