{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"44ddabe0-ebd2-5e79-ac61-a6e703b09744","task_key":"default--test--1814","task_revision_id":"1","upstream_id":"1814","short_description":"A man was at a party where he observed a female guest who had passed out and was…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"[\\\"Yes, the act of taking someone's belongings without their consent while they are unconscious is considered robbery.\\\", 'Yes, robbery does not require that the victim be subdued by force or be placed in fear.', 'No, this was not robbery because the victim was not aware of the crime when it occurred.', \\\"Yes, robbery is the unlawful taking of property from another's person by intimidation or force.\\\", 'No, this was not robbery because there was no intimidation, and indeed no force, against an unconscious victim.', 'No, this was not robbery because the man was a guest at the party.', 'Yes, robbery can occur in any location, including a private home.', 'No, this was not robbery because it occurred in a private home and not in a business establishment or on the street.', \\\"No, this was not robbery because the man didn't use a weapon.\\\"]\",\"question\":\"A man was at a party where he observed a female guest who had passed out and was unconscious and alone in a bedroom of the house. The man locked the door and quietly took the female's purse, emptied all of the money into his pockets, and removed the jewelry from her person. He was apprehended and charged with robbery. Can he be convicted of that crime?\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}