{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"d3095530-c766-5dc0-bc45-b1c7305dfe07","task_key":"default--test--941","task_revision_id":"1","upstream_id":"941","short_description":"After waiting in line for two hours to gain entry into a popular nightclub, a…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['not guilty, because he was coerced into committing the act.', 'guilty, because he did not attempt to alert authorities before throwing the explosives.', 'not guilty, because he was acting under immediate threat to his life.', 'guilty, because they were killings that occurred during the commission of an inherently dangerous felony.', 'not guilty, because the defendant was justified under the circumstances.', 'not guilty, because duress is a defense to arson.', 'guilty, as the crime was premeditated and caused multiple deaths.', 'guilty, because he physically committed the act of throwing the explosives.', 'not guilty, because he was not the primary actor in the crime.']\",\"question\":\"After waiting in line for two hours to gain entry into a popular nightclub, a man was denied admission because his attire failed to conform to the club's dress code. When he was refused admittance, the man angrily shouted to the club's doorman, \\\"You'll be sorry for this. After I'm through, this club will be reduced to rubble. \\\" Later that same evening, the man returned to the club with two explosives in his possession. He noticed the defendant leaving the club and followed him into a nearby parking lot. As the defendant was about to enter his car, the man grabbed him, pointed a gun and said, \\\"Follow me, you punk, or I'll blow your brains out. \\\" The man led the defendant to the rear of the club, handed him the explosives, and directed him to throw them through an open window of the club. In fear of his life, the defendant tossed the explosives into the club, causing an inferno that killed 25 people. In this jurisdiction, the applicable arson statute includes all buildings. If the defendant is charged with felony murder for the death of the people in the club, he will most likely be found\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}