{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"4018727a-ef95-5d6c-8005-e40c77eb09c1","task_key":"default--test--1213","task_revision_id":"1","upstream_id":"1213","short_description":"A neighborhood homeowners' association hired a security company to provide an…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['prevail, because the guard did not give a warning that he had a gun and was about to shoot.', \\\"not prevail, because the homeowner's association approved the use of force in such situations.\\\", 'prevail, because the guard should have called the police instead of taking matters into his own hands.', 'not prevail, because the guard acted within his rights to protect property within the neighborhood.', \\\"prevail, because the guard used unreasonable force to protect the homeowner's property.\\\", 'not prevail, because the young man was engaged in theft when he was shot.', \\\"not prevail, because the injury was a result of the young man's own criminal activity.\\\", \\\"not prevail, because the young man was trespassing on the homeowner's property.\\\"]\",\"question\":\"A neighborhood homeowners' association hired a security company to provide an armed guard to patrol the neighborhood. One evening, the guard saw a young man stealing a rake that a homeowner in the neighborhood had left outside in his yard. The guard ran toward the young man. Seeing the guard, the young man dropped the rake and began to flee. Believing that the young man was about to escape, the guard pulled out his gun and fired a shot at the young man. The bullet struck the young man in the arm. If the young man asserts a claim against the security company for damages for his injuries, the young man will\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}