{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"962a84b1-9415-5217-8afc-50950ba5c1a0","task_key":"default--test--5035","task_revision_id":"1","upstream_id":"5035","short_description":"With which of the following techniques were the native navigators of the Pacific…","config":"default","split":"test","body":"{\"category\":\"history\",\"options\":\"['understanding wave patterns', 'cloud patterns', 'magnetic orientation to the poles', 'bird flight paths', 'currents and wind patterns', 'celestial navigation using stars', 'orientation by the position of the sun', \\\"orientation by the moon's position\\\"]\",\"question\":\"With which of the following techniques were the native navigators of the Pacific NOT familiar?\",\"src\":\"ori_mmlu-prehistory\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}