{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"b231783c-68d9-5c64-9643-f75aac738aab","task_key":"default--test--8942","task_revision_id":"1","upstream_id":"8942","short_description":"Assuming we are underground, and the only thing we can observe is whether a…","config":"default","split":"test","body":"{\"category\":\"math\",\"options\":\"['[0, 1, 0]', '[1, 1, 0]', '[0, 0, 0]', '[0, 0, 1]', '[0, 1, 1]', '[1, 1, 1]', '[1, 0, 0]', '[1, 0, 1]']\",\"question\":\"Assuming we are underground, and the only thing we can observe is whether a person brings an umbrella or not. The weather could be either rainy or sunny. Assuming the P(rain)=0.6 and P(sunny)=0.4. Assuming the weather on day $k$ is dependent on the weather on day $k-1$. We can write the transition probability as P(sunny $\\\\mid$ sunny) = P(rain $\\\\mid$ rain) = 0.7. The person has 60\\\\% chance to bring an umbrella when the weather is rainy, and 40\\\\% chance to bring an umbrella when the weather is sunny, i.e. P(umbrella $\\\\mid$ rain) = 0.6 and P(umbrella $\\\\mid$ sunny) = 0.4. If we observe that the person (1) brought an umbrella on day 1, (2) did not bring an umbrella on day 2, (3) brought an umbrella on day 3.  What are the most likely weather from day 1 to day 3? Return the answer as a list of binary values, where 1 represents rain and 0 represents sunny.\",\"src\":\"theoremQA-Math\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}