{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"456f2cbe-3acc-502e-91e4-c9ab7ba98771","task_key":"default--test--10608","task_revision_id":"1","upstream_id":"10608","short_description":"Suppose the codeword that we use to describe a random variable X always starts…","config":"default","split":"test","body":"{\"category\":\"computer science\",\"options\":\"['[7, 8, 9, 73, 83, 93]', '[7, 8, 9, 71, 81, 91]', '[7, 8, 9, 710, 810, 910]', '[7, 8, 9, 70, 80, 90]', '[7, 8, 9, 700, 800, 900]', '[7, 8, 9, 74, 84, 94]', '[7, 8, 9, 77, 87, 97]', '[7, 8, 9, 75, 85, 95]', '[7, 8, 9, 72, 82, 92]', '[7, 8, 9, 76, 86, 96]']\",\"question\":\"Suppose the codeword that we use to describe a random variable X always starts with a symbol chosen from the set {7, 8, 9} , followed by binary digits {0, 1}. Thus we have a ternary code for the first symbol and binary thereafter. Give the optimal uniquely decodeable code (minimum expected number of symbols) for the probability distribution $p = (16/69, 15/69, 12/69, 10/69, 8/69, 8/69)$.\",\"src\":\"theoremQA-EECS\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}