{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"385bd6bb-f047-55ae-99c9-453350dbf3ad","task_key":"default--test--1327","task_revision_id":"1","upstream_id":"1327","short_description":"A state has a statute prohibiting the sale of cigarettes to any person under the…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['guilty, because the defendant was in charge of the store at the time of the violation.', \\\"not guilty, because the defendant's employer is vicariously liable for the violation that occurred.\\\", 'not guilty, because the defendant did not physically sell the cigarettes to the girl.', 'not guilty, because the defendant did not receive any money for the cigarettes.', 'not guilty, because the defendant was unconscious.', \\\"not guilty, because the girl took the cigarettes without the defendant's knowledge.\\\", 'guilty, because the 15-year-old girl should have been denied service.', 'guilty, because the defendant left the cigarettes in an accessible place.', \\\"guilty, because the defendant's intoxication was voluntaiy.\\\"]\",\"question\":\"A state has a statute prohibiting the sale of cigarettes to any person under the age of 17. Violation of the statute is a misdemeanor punishable by a fine not less than $500 or more than $5,000 The state courts have interpreted this statute as a public welfare offense requiring no particular mental state for its violation. As such, this is a strict liability crime whereby culpability is imposed on a defendant for doing the proscribed act. The defendant, a recovering alcoholic, worked as a cashier at a drug store. Late one night while the defendant was working alone, he got the urge to have a drink. He opened a bottle of rum that the store sold and soon began drinking. A short time later, the defendant became inebriated and passed out. Not long thereafter, a 15-year-old girl entered the store to purchase a package of cigarettes. She took a box off the shelf and went to the cashier's counter to pay for them. Seeing the defendant lying on the floor, unconscious, the girl left the exact price for the cigarettes on the countertop and left the store. If the defendant is prosecuted for violating the state statute, he should be found\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}