{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"d723e42a-a961-5528-8723-b00126aa018b","task_key":"default--test--1198","task_revision_id":"1","upstream_id":"1198","short_description":"A man informed the police that he wanted to confess to a murder. After they read…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['Yes, because the man was suffering from hallucinations at the time of his confession.', 'No, there was no police misconduct or overreaching, and as a result the confession is voluntary.', 'No, a confession is admissible when accompanied by actions by the defendant that prove the truth of the statements.', 'No, because the man insisted initially that he was acting voluntarily.', 'Yes, because the man later retracted his confession.', 'Yes, the police overreached by not having him mentally evaluated before questioning him.', 'Yes, he was insane and could not have given a voluntary confession.', \\\"No, because the man led the police to the victim's body which supports his confession.\\\", 'No, because the police read the man his constitutional rights before he confessed.']\",\"question\":\"A man informed the police that he wanted to confess to a murder. After they read him his constitutional rights, he insisted that he was acting voluntarily. He then took them to the scene of the murder where they found the victim's body. By the next day, the man claimed that \\\"voices\\\" had ordered him to confess and he didn't act voluntarily. A physician verified that the man suffered from hallucinations that impaired his volitional abilities, preventing a voluntary confession. Will the court suppress the statements?\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}