{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"7f617a0f-bfe9-5c82-b77a-89998af9931a","task_key":"default--test--1093","task_revision_id":"1","upstream_id":"1093","short_description":"George was charged with the murder of a well- known criminal. The victim was…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['leading the witness.', 'privileged communication.', 'speculative.', 'opinion testimony.', 'character evidence.', 'irrelevant.', 'an assertion by the witness.', 'self-serving.', 'circumstantial evidence.']\",\"question\":\"George was charged with the murder of a well- known criminal. The victim was found shot to death in his car. The only witness to the shooting was the victim's pet parrot, which was found hovering over the victim's body repeating over and over, \\\"George don't. . . George don't. \\\"To prove the identity of the victim's killer, the prosecution has a witness testify that about the time of the murder, he was in the area and heard the victim's parrot calling George's name. The best objection to the testimony is that it is\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}