{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"891c0453-e171-5631-9af3-24ce6d6dbb61","task_key":"default--test--1437","task_revision_id":"1","upstream_id":"1437","short_description":"A man hosted several friends at his home for dinner. Two of the guests got into…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"[\\\"No, because the host's use of force was excessive under the circumstances.\\\", \\\"Yes, because the host's actions were justified under the doctrine of necessity.\\\", \\\"No, because the host's use of a weapon escalated the situation.\\\", 'No, because the host should have called the police instead of intervening.', 'No, because the host had a duty to retreat before using force.', 'Yes, because the host has the privilege of host immunity.', 'No, because the host escalated the conflict by getting involved.', 'Yes, because the host was acting in self-defense.', 'Yes, because the injured guest was committing a crime when the host intervened.', 'Yes, because the host had the privilege of \\\"defense of others.\\\"']\",\"question\":\"A man hosted several friends at his home for dinner. Two of the guests got into an argument, and one of them got up from the table and lunged at the other with a large hunting knife. The host stepped in with a baseball bat and hit the attacker in the chest with it, preventing the imminent contact by the attacker with his target. The blow knocked the attacker down and caused several broken ribs. The injured guest sued the host for aggravated assault and battery. Is the court likely to grant the host's motion to dismiss the claim for failure to state a sufficient claim for relief?\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}