{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"dfe3e043-9f16-5503-8996-09aad28dd5e1","task_key":"default--test--10415","task_revision_id":"1","upstream_id":"10415","short_description":"Consider the following grammar.","config":"default","split":"test","body":"{\"category\":\"computer science\",\"options\":\"['III only and not II', 'II only and not III', 'I only', 'I and II only', 'III only', 'Neither I, II, nor III', 'II and III only', 'II only', 'I and III only', 'I, II, and III']\",\"question\":\"Consider the following grammar.\\nS → ( S )\\nS → x\\nWhich of the following statements is (are) true?\\nI. The grammar is ambiguous.\\nII. The grammar is suitable for top-down parsing.\\nIII. The grammar is suitable for bottom-up parsing.\",\"src\":\"ori_mmlu-college_computer_science\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}