{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"0d61d45b-c29f-5e99-afbb-165d49c691b5","task_key":"default--test--1540","task_revision_id":"1","upstream_id":"1540","short_description":"A father and a son were indicted and charged with bank robbery. They allegedly…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['admitted against the father only, because the son recanted his confession.', 'not admitted against the son, but admitted against the father as it implicates him in the crime.', 'admitted against both the son and the father, because the son implicated the father in the crime.', 'not admitted against the son or the father, because the son was under duress when he confessed.', 'admitted against the son only, because it is his confession.', 'admitted against both the son and the father, because they were jointly charged.', 'not admitted against the son or the father, because the confession is hearsay not within any recognized exception.', 'not admitted against the son or the father, because the son recanted the confession.', 'admitted against both the son and the father, because they were engaged in a conspiracy.', 'admitted against the son only.']\",\"question\":\"A father and a son were indicted and charged with bank robbery. They allegedly participated together in robbing a bank in their hometown. After the son was arrested and given Miranda warnings, he confessed and admitted that he and his father took part in the bank robbery. The prosecution decided to charge the father and son in the same indictment and prosecute them together in a single trial. At trial, the son took the stand and recanted his earlier confession by denying participation in the robbery. The prosecution now seeks to introduce into evidence the son's earlier confession that was videotaped by the police. The son and the father both object to its admissibility. The confession should be\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}