{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"65174509-31f0-5d50-9e67-3ba853cfa9e8","task_key":"default--test--4908","task_revision_id":"1","upstream_id":"4908","short_description":"Inca architecture is noted for its:","config":"default","split":"test","body":"{\"category\":\"history\",\"options\":\"['focus on creating intricate geometric patterns.', 'construction of structures exclusively on hilltops.', 'employment of advanced hydraulic systems.', 'incorporation of wood, earth, and stone.', 'reliance on natural materials such as leaves and vines.', 'extensive use of glass and metal.', 'use of the vaulted arch.', 'unprecedented size and sophistication.', 'use of circular designs in most structures.', 'precise jigsaw-like stone masonry.']\",\"question\":\"Inca architecture is noted for its:\",\"src\":\"ori_mmlu-prehistory\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}