{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"9ecc7352-876e-59ed-a277-4fc08793af50","task_key":"default--test--2520","task_revision_id":"1","upstream_id":"2520","short_description":"According to the Standards for Educational and Psychological Testing, when…","config":"default","split":"test","body":"{\"category\":\"psychology\",\"options\":\"['exclude the potential effects of situational and demographic variables from consideration', 'Avoid discussing the test results with the tester to prevent performance anxiety', 'Give more weight to the lowest score as it indicates areas for improvement', \\\"Always associate test results with the tester's demographic information for context\\\", 'use descriptive labels, because such labels readily communicate specific meanings to most people', 'Neglect the confidence interval, since the obtained score represents the true score', 'focus attention on the confidence interval of a score rather than on the obtained score itself', 'stress the obtained score, since it is the best estimate of the true score', 'Only interpret the scores in comparison to the average result of the group', \\\"Consider only the highest obtained score as it demonstrates the tester's potential\\\"]\",\"question\":\"According to the Standards for Educational and Psychological Testing, when interpreting and reporting test results, one should\",\"src\":\"ori_mmlu-professional_psychology\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}