{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"576a7adf-f443-5bba-a6d2-6f8cb6633d3c","task_key":"default--test--1238","task_revision_id":"1","upstream_id":"1238","short_description":"A defendant is on trial for kidnapping. The victim has testified that one of the…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"[\\\"Yes, it's relevant to show the defendant's nickname.\\\", \\\"No, because it's irrelevant to the case.\\\", 'No, because the jail employee is not a reliable source of information.', 'Yes, to corroborate the truthfulness of the victim.', 'Yes, as circumstantial evidence that the defendant was one of the kidnappers.', 'No, because it is substantially more prejudicial than probative.', 'Yes, because it provides character evidence about the defendant.', \\\"No, because it's an opinion not based on the employee's personal knowledge.\\\", \\\"Yes, because it's a statement made by the defendant.\\\", 'No, because it is hearsay not within any exception. exception.']\",\"question\":\"A defendant is on trial for kidnapping. The victim has testified that one of the kidnappers referred to the other as \\\"Speed.\\\" The prosecutor calls a jail employee to testify that, while the defendant was in jail awaiting trial, other inmates addressed the defendant as \\\"Speed.\\\" Is the jail employee's testimony admissible?\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}