{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"258a5228-6d27-5ec5-9bf7-72a76b1cdcca","task_key":"default--test--7888","task_revision_id":"1","upstream_id":"7888","short_description":"Draw one card at random from a standard deck of cards. The sample space $S$ is…","config":"default","split":"test","body":"{\"category\":\"math\",\"options\":\"['$\\\\\\\\frac{14}{52}$', '$\\\\\\\\frac{12}{52}$', '$\\\\\\\\frac{13}{52}$', '$\\\\\\\\frac{15}{52}$', '$\\\\\\\\frac{10}{52}$', '$\\\\\\\\frac{7}{52}$', '$\\\\\\\\frac{11}{52}$', '$\\\\\\\\frac{9}{52}$', '$\\\\\\\\frac{8}{52}$', '$\\\\\\\\frac{16}{52}$']\",\"question\":\"Draw one card at random from a standard deck of cards. The sample space $S$ is the collection of the 52 cards. Assume that the probability set function assigns $1 / 52$ to each of the 52 outcomes. Let\\n$$\\n\\\\begin{aligned}\\nA & =\\\\{x: x \\\\text { is a jack, queen, or king }\\\\}, \\\\\\\\\\nB & =\\\\{x: x \\\\text { is a } 9,10, \\\\text { or jack and } x \\\\text { is red }\\\\}, \\\\\\\\\\nC & =\\\\{x: x \\\\text { is a club }\\\\}, \\\\\\\\\\nD & =\\\\{x: x \\\\text { is a diamond, a heart, or a spade }\\\\} .\\n\\\\end{aligned}\\n$$\\nFind $P(A)$\",\"src\":\"scibench-stat\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}