{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"02cb30e6-25e8-54da-a785-add31b818586","task_key":"default--test--1122","task_revision_id":"1","upstream_id":"1122","short_description":"Which of the following most accurately describes Hart's response to Fuller's…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['The Nazi law in question was validly enacted.', 'The Nazi rule of recognition was unclear.', \\\"Hart agreed with Fuller's argument.\\\", 'The court misunderstood the legislation.', 'Fuller misconstrued the purpose of the law.', 'Hart argued that the law should be disregarded due to its origin.', 'Hart claimed the law was invalid due to a lack of enforcement.', \\\"Hart contended Fuller's argument was irrelevant to the validity of the law.\\\", 'Hart believed the Nazi law was morally justified.', 'Hart suggested that the Nazi law was not truly a law.']\",\"question\":\"Which of the following most accurately describes Hart's response to Fuller's argument concerning the invalidity of Nazi law?\",\"src\":\"ori_mmlu-jurisprudence\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}