{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"4f68cda1-0094-54d3-9a72-3ce22c184110","task_key":"default--test--1164","task_revision_id":"1","upstream_id":"1164","short_description":"A defendant is on trial for robbery. The defendant's attorney plans to call a…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"[\\\"denied, because the conviction is relevant to the witness's credibility\\\", 'denied, because the credibility of a witness may be attacked by any party.', \\\"granted, because the prejudice to the defendant substantially outweighs the probative value of admitting the witness's conviction.\\\", 'denied, because the conviction is a matter of public record', 'granted, as a matter of right.', 'denied, because the defense attorney did not file the motion in a timely manner', 'denied, because a period of less than 10 years has elapsed since the date of conviction.', \\\"granted, because the witness's past has no bearing on the defendant's case\\\", 'granted, because the conviction is not related to the current case']\",\"question\":\"A defendant is on trial for robbery. The defendant's attorney plans to call a witness. Nine years earlier, the witness had been convicted of misdemeanor battery. The defendant's attorney has filed a motion disallowing the prosecution from questioning the witness regarding this prior conviction. The motion will likely be\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}