{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"ee3d3b7b-6476-5bc1-8ca3-ba1bfc546244","task_key":"default--test--1193","task_revision_id":"1","upstream_id":"1193","short_description":"A man was in jail after being arrested for burglary. When the police attempted…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"[\\\"No, because the man's prior invocation of his Miranda rights did not preclude the later interrogation.\\\", 'Yes, because the man was under duress while being interrogated in prison.', 'Yes, because the man had earlier invoked his Miranda rights, and the police were not permitted to resume questioning, even after a time lapse of years.', 'Yes, because the man was not informed of his Miranda rights before the second interrogation.', 'No, because the man was serving a prison sentence, which nullifies his Miranda rights.', \\\"Yes, because the detective should have been informed of the man's earlier invocation of his Miranda rights.\\\", 'No, because the detective was unaware that the man had originally invoked his Miranda rights.', 'Yes, because the man was incarcerated, and his earlier invocation of his Miranda rights shielded him from further questioning until shielded him from further questioning until he was released.', 'No, because the new charge of homicide is unrelated to the original burglary charge.', 'No, because the man willingly waived his Miranda rights during the second interrogation.']\",\"question\":\"A man was in jail after being arrested for burglary. When the police attempted to question him, the man invoked his Miranda rights and refused to answer any questions. The man was subsequently tried, convicted, and sentenced to a prison term for the burglary. Three years later, while the man was serving his prison sentence for the burglary, a police detective from a nearby town questioned him about an unsolved homicide. The detective did not know that the man had invoked Miranda at an earlier time. that the man had invoked Miranda at an earlier time. The man waived his Miranda rights and made several incriminating statements to the detective. When he was later charged with the homicide, the man moved to suppress these statements, claiming that his earlier refusal to waive his Miranda rights should have been honored. Should the court suppress the statements?\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}