{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"d9ab7171-ab26-548b-a705-4ef263aa6862","task_key":"default--test--1896","task_revision_id":"1","upstream_id":"1896","short_description":"An experienced equestrian rider sued a woman, claiming that her horse kicked and…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['inadmissible, because it is hearsay evidence.', 'inadmissible as an improper form of impeachment.', 'inadmissible as substantive evidence to show that the horse must have been provoked.', \\\"inadmissible, because the woman's testimony contradicts the rider's account.\\\", \\\"inadmissible as beyond the scope of the woman's direct testimony.\\\", \\\"admissible as it provides evidence of the horse's general behavior.\\\", \\\"inadmissible, because the woman's testimony is biased.\\\", 'admissible as it is relevant to the case.', \\\"admissible, because it can refute the rider's claim.\\\"]\",\"question\":\"An experienced equestrian rider sued a woman, claiming that her horse kicked and injured the rider. At trial, the rider testified that he was attending a horse show when, without provocation, he was kicked by a gray stallion with a black tail. The rider then called the woman as an adverse witness and asked her the following question: \\\"Isn't it true that you are the owner and trainer of a gray stallion with a black tail?\\\" After answering affirmatively, the woman was not asked any further questions on direct examination. Whereupon in cross- examination, the woman was then asked, \\\"Isn't it true that your horse is very gentle and would not kick anyone unless provoked?\\\"Upon objection by the rider's attorney, the rider's strongest argument would be that the woman's proffered testimony is\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}