{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"f75c0236-bc7f-54b2-ae67-f16854dbff12","task_key":"default--test--1368","task_revision_id":"1","upstream_id":"1368","short_description":"A defendant was arrested at an airport when the small suitcase he was carrying…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"[\\\"permissible, because the defendant's past addiction could provide context for the possession charge.\\\", \\\"permissible, because the defendant's physical condition is a matter of public record.\\\", \\\"objectionable, because it is an invasion of the defendant's personal privacy.\\\", 'objectionable, because the defendant has a privilege against self-incrimination.', \\\"objectionable, because the defendant's past addiction is protected health information.\\\", \\\"objectionable, because the defendant's past addiction is not relevant to the current charges.\\\", \\\"permissible, because such evidence is relevant to the defendant's credibility.\\\", 'permissible, because the defendant waived his privilege against self-incrimination by taking the stand.', 'objectionable, because the probative value is substantially outweighed by the danger of unfair prejudice.', \\\"permissible, because the defendant's past addiction could imply a motivation for possession.\\\"]\",\"question\":\"A defendant was arrested at an airport when the small suitcase he was carrying was found to contain heroin. The defendant, who did not challenge the legality of the airport search, was subsequently prosecuted for possession of heroin. At trial, the defendant testified on his own behalf and said that the suitcase belonged to his girlfriend who was accompanying the defendant when he was arrested. The girlfriend died in a skydiving accident two weeks before the defendant's trial. Moreover, the defendant testified that although he was a former heroin addict, he had not used any heroin in the past three years. On cross-examination, the prosecuting attorney asked the defendant to roll up the sleeves of his shirt and exhibit his arms to see if there were any needle marks. This request is\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}