{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"05ab0b00-dcb6-51fd-971b-6549cacbe4b3","task_key":"default--test--1756","task_revision_id":"1","upstream_id":"1756","short_description":"A victim was standing on a street corner waiting to make a drug buy. The drug…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['the jury, after a preliminary determination by the judge.', 'the jury, without a preliminary determination by the judge.', 'the judge, only after consultation with a legal expert.', 'the judge, with instruction to the jury that it may consider the dying declaration if it determines that the victim knew he was dying.', 'the police department that conducted the initial investigation.', 'the jury, with instruction from the judge that it may disregard the dying declaration if it determines the victim was under duress.', 'the jury, with a requirement to cross-examine the statement with other presented evidence.', 'the judge, without assistance from the jury.', 'a third-party mediator, to prevent any bias.']\",\"question\":\"A victim was standing on a street corner waiting to make a drug buy. The drug dealer was always late, so the victim continued to wait outside in the freezing cold weather because he needed a quick fix. Suddenly a car rounded the street corner and a shot rang out. The victim slumped over and fell to the sidewalk. Immediately after being shot and before he died, the victim exclaimed, \\\"The defendant did it!\\\" The defendant was indicted and is now on trial for the victim's murder. Based on these facts alone, the admissibility of the victim's statement should be determined by\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}