{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"4b7a2a41-5e8c-5d81-97ea-92e7200b09dc","task_key":"default--test--1450","task_revision_id":"1","upstream_id":"1450","short_description":"A ski resort had a lift that carried customers from the bottom to the top of the…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['intentional infliction of emotional distress.', 'assault.', 'false imprisonment.', 'invasion of privacy.', 'negligence.', 'battery.', 'breach of contract.', 'defamation.', 'no cause of action.', 'trespass.']\",\"question\":\"A ski resort had a lift that carried customers from the bottom to the top of the ski slope. One afternoon, the lift suddenly broke down because of a power failure in the area. A customer was suspended in his lift chair, one-third of the way up the hill and 50 feet above the ground. The customer remained on the lift for five hours until power was restored. He was then returned uninjured to the bottom of the hill. In a suit against the ski resort, the customer is likely to have action for\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}