{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"9d201d74-510c-52fe-a163-d4e54ca9f658","task_key":"default--test--1639","task_revision_id":"1","upstream_id":"1639","short_description":"A professional football player was seriously injured when the football helmet he…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['inadmissible, because the former employee is no longer authorized to speak on behalf of the manufacturer.', 'admissible as an admission.', 'inadmissible, because of the public policy behind taking safety precautions.', 'admissible as evidence that the manufacturer knew the helmet was defective.', \\\"admissible, because the former employee has firsthand knowledge of the manufacturer's actions.\\\", 'inadmissible, because the testimony is prejudicial to the manufacturer.', \\\"admissible as direct evidence that the player's injury was caused by the defective design of the football helmet.\\\", 'inadmissible, because the testimony is hearsay.', 'inadmissible, because the testimony is irrelevant to the case.', \\\"admissible as circumstantial evidence that the player's injury was traceable to the defective design of the football helmet.\\\"]\",\"question\":\"A professional football player was seriously injured when the football helmet he was wearing shattered as he was being tackled during a game. The player subsequently asserted a claim against the manufacturer of the helmet to recover damages for his injury. At trial, the player's attorney calls a former employee of the manufacturer to testify that three weeks after the player suffered his injury, the manufacturer devised a new design for its football helmets. The former employee's testimony is\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}