{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"041b2916-c6a8-57c5-ba78-881ebfb0ad46","task_key":"default--test--1753","task_revision_id":"1","upstream_id":"1753","short_description":"A man allowed his friend to borrow a debit card to buy a computer. When the…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['No, because the evidence was sufficient to support a verdict of intentional murder.', 'No, because the evidence was sufficient to support a verdict of depraved-heart murder.', 'Yes, because the evidence showed that the discharge of the gun was accidental.', 'Yes, because the man was acting in self-defense after being attacked by the bystander.', \\\"Yes, because the evidence showed that the man was provoked by his friend's refusal to return the debit card.\\\", 'No, because the evidence was sufficient to support a verdict of premeditated murder.', 'No, because the evidence was sufficient to support a verdict of felony murder.', 'Yes, because the evidence showed that the man did not intentionally pull the trigger.', \\\"Yes, because the evidence showed that the man's intent was not to kill the friend, but to take property from him under a claim of right.\\\"]\",\"question\":\"A man allowed his friend to borrow a debit card to buy a computer. When the friend refused to return the card during a later conversation in a bar, the man pointed a handgun at the friend, held his finger on the trigger, and said, \\\"Give it back!\\\" A bystander screamed and threw a beer bottle at the man. The bottle struck the man's hand and caused the gun to discharge, killing the friend instantly. The man was convicted of murder in a jurisdiction that follows the common law of homicide. On appeal, he argues that the evidence, which essentially set out the facts above, was insufficient to support a verdict of murder. Should the man's conviction be reversed?\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}