{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"f12a7afd-889a-58e0-9e69-b349e15c8d33","task_key":"default--test--906","task_revision_id":"1","upstream_id":"906","short_description":"Two men were told by the defendant that a medical devices company was now using…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['a direct participant.', 'an accessory after the fact.', 'a principal in the second degree.', 'an accomplice.', 'an indirect participant.', 'an accessory before the fact.', 'not respbnsible.', 'a primary offender.', 'a principal in the first degree.', 'a conspirator.']\",\"question\":\"Two men were told by the defendant that a medical devices company was now using the old warehouse on the river for storing its supply of defibrillators. Late one night, the two men broke into the warehouse and proceeded to load the large crates of defibrillators onto their truck. As they were leaving, one of the men inadvertently threw his cigarette butt into a refuse pile of old cardboard boxes and papers. Soon afterward, the refuse ignited into a small fire. Although the men had time to douse the fire without any danger to themselves, they did not do so. Instead, they fled. Meanwhile, the fire quickly spread, engulfing the entire warehouse in flames. If the defendant is later charged as a co-conspirator, in all likelihood he would be held responsible as\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}