{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"ac17e7e0-cafe-5d9a-9ccd-a40f0d84ee23","task_key":"default--test--1027","task_revision_id":"1","upstream_id":"1027","short_description":"Jones is arrested for driving under the influence of a controlled substance,…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"[\\\"Admissible because the question goes to Jones' credibility as a witness.\\\", 'Admissible, as it relates to the case at hand.', 'Admissible, so long as the prosecutor produces a certified copy of the conviction.', \\\"Admissible, because it showcases Jones' character.\\\", 'Not admissible, because specific instances of conduct cannot be proved with extrinsic evidence.', 'Not admissible, as it is considered hearsay.', \\\"Admissible, because it demonstrates Jones' dishonesty.\\\", 'Not admissible because it is irrelevant.', 'Not admissible, because it is prejudicial to the defendant.']\",\"question\":\"Jones is arrested for driving under the influence of a controlled substance, marijuana. The prosecution wishes to introduce a videotape from a pot party Jones left just before his arrest. The recording shows Jones exhaling a giant cloud of smoke and stating, \\\"that's really great weed, man.\\\" Jones testifies on his own behalf, \\\"I'm an honest person. I absolutely did not drive erratically.\\\" On cross-examination, the prosecutor asks, \\\"Isn't it true you lied in your divorce trial last year?\\\" If Jones' attorney objects, the court will likely hold the prosecutor's question is\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}