{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"4116e9c3-ffc8-5dcf-b8f1-73ca5a8d140a","task_key":"default--test--1112","task_revision_id":"1","upstream_id":"1112","short_description":"A homeowner awoke one night and observed a thief stealing chickens from his…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['Yes, because the thief was committing a crime, and the homeowner had a right to defend his property.', 'Yes, he was authorized to use deadly force where he reasonably believed that the thief had a gun and would harm him.', 'No, because the homeowner should have called the authorities instead of taking matters into his own hands.', 'Yes, because the homeowner believed he saw a gun, and thus had reason to fear for his life.', 'No, because a homeowner can never legally use deadly force to stop a theft.', 'Yes, because the homeowner was in his own home and had the right to protect himself and his property.', 'Yes, because a dwelling owner can always use deadly force to protect his property and person from an illegal trespasser or thief.', 'No, because the homeowner used excessive force in response to the situation.', 'No, because it is not reasonable to use deadly force to prevent the theft of the chickens.', 'No, because the thief was leaving the property and no longer posed a threat.']\",\"question\":\"A homeowner awoke one night and observed a thief stealing chickens from his backyard chicken coop. He thought he saw a gun but could not tell for sure. While the thief was exiting with a sack of chickens, the homeowner got his hunting rifle and shot him dead. The authorities charged the homeowner with homicide. At the trial, he defended on the basis that he was in fear for his safety, and was legally authorized to use deadly force to defend his dwelling and person from the illegal threat. Is the defense to homicide valid under these facts?\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}