{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"3da7458d-88f7-58d7-8cd4-421c7548fbb1","task_key":"default--test--1955","task_revision_id":"1","upstream_id":"1955","short_description":"Late one night, co-defendants broke into a warehouse and proceeded to load the…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['burglary and larceny.', 'burglary only.', 'burglary and arson.', 'larceny and arson.', 'burglary, arson, and criminal negligence.', 'burglary, larceny, and arson.', 'arson only.', 'larceny only.', 'larceny, arson, and criminal negligence.']\",\"question\":\"Late one night, co-defendants broke into a warehouse and proceeded to load the large crates of appliances onto their truck. As they were leaving, one of the co-defendants inadvertently threw his cigarette butt into a refuse pile of old cardboard boxes and papers. Soon afterward, the refuse ignited into a small fire. Although the co-defendants had time to douse the fire without any danger to themselves, neither did so. Instead, they climbed into the truck and fled. Meanwhile, the fire quickly spread, engulfing the entire warehouse in flames. At common law, the co-defendants should be found guilty of\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}