{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"3fec9c3a-4ab5-57c4-b86c-79bb4c5d3dc1","task_key":"default--test--2517","task_revision_id":"1","upstream_id":"2517","short_description":"Which of the following is the best example of constant error","config":"default","split":"test","body":"{\"category\":\"psychology\",\"options\":\"['A subject consistently provides correct answers to every test item', 'A psychometrician consistently underestimates IQ scores by 10 points', 'A test item is answered correctly by every student in a given class', 'A subject consistently provides wrong answers to every test item', 'A test-retest reliability coefficient is high for several samples', 'A subject inconsistently provides wrong answers to every test item', 'A psychometrician inconsistently overestimates IQ scores', 'A test item is answered incorrectly by every student in a given class', 'A test-retest reliability coefficient is low for several samples']\",\"question\":\"Which of the following is the best example of constant error\",\"src\":\"ori_mmlu-professional_psychology\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}