{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"1c167562-df97-5a7b-b94a-3ece153334a6","task_key":"default--test--4931","task_revision_id":"1","upstream_id":"4931","short_description":"Based on archaeological evidence, which of the following is accurate regarding…","config":"default","split":"test","body":"{\"category\":\"history\",\"options\":\"['The occurrence of warfare is uncommon and occurs only with the concentration of wealth.', 'The occurrence of warfare decreases as people increasingly invest time in the production of art.', 'The occurrence of warfare and concentration of wealth are both uncommon features.', 'The occurrence of warfare is directly proportional to the concentration of wealth.', 'The concentration of wealth is a common feature.', 'The production of art decreases as people increasingly invest time in warfare.', 'The concentration of wealth is uncommon and occurs only occasionally.']\",\"question\":\"Based on archaeological evidence, which of the following is accurate regarding civilization in general?\",\"src\":\"ori_mmlu-prehistory\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}