{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"c3acd1be-431a-5af1-a673-dc3a8c058cee","task_key":"default--test--10416","task_revision_id":"1","upstream_id":"10416","short_description":"An Euler circuit of an undirected graph is a circuit in which each edge of the…","config":"default","split":"test","body":"{\"category\":\"computer science\",\"options\":\"['A complete graph with 11 vertices', 'III only', 'I and II', 'None of the above', 'II and III', 'All of the above', 'I only', 'II only', 'I and III', 'A tree with 12 vertices']\",\"question\":\"An Euler circuit of an undirected graph is a circuit in which each edge of the graph appears exactly once. Which of the following undirected graphs must have an Euler circuit?\\nI. A complete graph with 12 vertices\\nII. A complete graph with 13 vertices\\nIII. A tree with 13 vertices\",\"src\":\"ori_mmlu-college_computer_science\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}