{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"580677e3-e738-53a3-a61b-ed62f09c68ff","task_key":"default--test--4909","task_revision_id":"1","upstream_id":"4909","short_description":"what is one of the important differences between the middle and upper…","config":"default","split":"test","body":"{\"category\":\"history\",\"options\":\"['a profusion of stone tool traditions, indicating a change from temporal and geographic homogeneity to greater diversity and variability', 'a gradual decline in the use of stone hand axes and tools, indicating a change to more flexible and workable materials such as wood and bone', 'decreased use of imported raw materials and increased use of whatever was locally available', 'smaller sites, indicating a change from large roving bands of hunters to the earliest family groups and households', 'a gradual increase in the use of metal tools, indicating a shift away from stone tools', 'no significant changes in tool usage, indicating a continuity in lifestyle and technology', 'larger sites, indicating a change from small family groups to larger bands of hunters', 'increased use of imported raw materials and decreased use of local resources', 'increased use of stone hand axes and tools, indicating a change to less flexible materials', 'a decrease in stone tool traditions, indicating a shift to uniformity and less variability']\",\"question\":\"what is one of the important differences between the middle and upper paleolithic?\",\"src\":\"ori_mmlu-prehistory\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}