{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"96814e4e-53a6-5f44-8b07-1593cce296fb","task_key":"default--test--1586","task_revision_id":"1","upstream_id":"1586","short_description":"Two co-defendants were investment brokers at a bank. They had been employed by…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['Only the co-defendant who made the phone call is guilty of murder, but both are guilty of conspiracy to commit murder.', 'Both co-defendants are guilty of murder and conspiracy to commit murder.', 'Both co-defendants are guilty of conspiracy to commit murder, but not murder.', 'Both co-defendants are not guilty of murder and conspiracy to commit murder.', 'Both co-defendants are guilty of felony murder, but neither is guilty of conspiracy to commit murder.', 'Both co-defendants are guilty of felony murder, and only one is guilty of conspiracy to commit murder.', 'Only the co-defendant who placed the bomb is guilty of felony murder and conspiracy to commit murder.', 'Only the co-defendant who made the phone call is guilty of felony murder and conspiracy to commit murder.', 'Both co-defendants are guilty of conspiracy to commit murder, and only one is guilty of murder.', 'Only the co-defendant who placed the bomb is guilty of murder, but both are guilty of conspiracy to commit murder.']\",\"question\":\"Two co-defendants were investment brokers at a bank. They had been employed by the bank for ten years. One day, they were unexpectedly fired by the bank's new manager. Apparently, the co-defendants' investments had not been performing well in the economic downturn. The co-defendants, upset and humiliated by their firing, vowed to get back at the bank. As their revenge, they decided to plant a bomb in the bank and demand $2,000,000. After receiving the money, they would then reveal the location of the bomb and provide details for defusing it. The co-defendants agreed that the bank should be given adequate warning so that nobody would be injured. In accordance with their plan, the co-defendants consulted the Internet, where they found all the information they needed on how to build a bomb. They built the bomb and placed it in a supply closet. Shortly thereafter, they phoned the bank and made a demand for the money. They said a bomb was in the bank and that it would explode in 24 hours unless the money was paid. The bank refused to pay the money. Realizing that their plan had gone awry, they drove to the bank to attempt to defuse the bomb. As they were driving back to the bank, the bomb exploded, killing 30 people. If the co-defendants are charged with murder and conspiracy to commit murder, which of the following statements is correct?\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}