{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"ada9f20a-ad9e-562c-95f6-554b31e3243d","task_key":"default--test--951","task_revision_id":"1","upstream_id":"951","short_description":"A man was visiting his friend at the friend's cabin. The man decided to surprise…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['No, because the man did not intend to burn down the cabin and was not guilty of wanton and willful misconduct.', 'Yes, because the man purchased and placed the fireworks that caused the explosion.', 'No, because the man did not light the fireworks himself.', 'No, because the man never intended to hurt the friend.', \\\"Yes, because the man indirectly set fire to the friend's cabin.\\\", \\\"Yes, because the man's actions indirectly led to the destruction of the cabin.\\\", 'Yes, because the man should have known the fireworks would be a fire hazard.', \\\"No, because the man's actions were meant as a surprise, not a malicious act.\\\", \\\"Yes, because the injury to the friend and to his cabin was the natural result of the man's actions.\\\", \\\"No, because the friend's cigarette, not the man, caused the fire.\\\"]\",\"question\":\"A man was visiting his friend at the friend's cabin. The man decided to surprise the friend with some fireworks, so he purchased some fireworks and hid them under his friend's chair on the friend's porch. That evening, the friend went outside to smoke and dropped his unfinished cigarette next to his chair on the porch. The cigarette ignited the fireworks, and the fireworks immediately exploded, causing extensive damage to the friend and to his cabin. The man was arrested and is being prosecuted for arson. Will the man be found guilty of arson?\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}