{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"d27be014-de2f-53f4-9bca-f9c5e88746d6","task_key":"default--test--2715","task_revision_id":"1","upstream_id":"2715","short_description":"There are specific research designs and strategies used to gain the desired…","config":"default","split":"test","body":"{\"category\":\"psychology\",\"options\":\"['when reversal is not possible', 'when assignment is not possible', 'when selection is not possible', 'when there are not enough data available']\",\"question\":\"There are specific research designs and strategies used to gain the desired information. From your knowledge about the research designs and strategies, in which case is the MULTIPLE BASELINE DESIGN used?\",\"src\":\"ori_mmlu-professional_psychology\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}