{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"72dd73a5-dddc-5ea0-9457-6bbe535ce04d","task_key":"default--test--1887","task_revision_id":"1","upstream_id":"1887","short_description":"In which of the following situations would the defendant most likely be found…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['A defendant approached a man on the street and asked for his wallet. The man refused and the defendant threatened him with a knife. The man handed over his wallet and the defendant ran away.', 'A defendant was walking behind a shopper in a mall when he suddenly reached for her gold chain, pulled it from her neck, and ran away into the crowd. The shopper suffered a slight cut on her neck where the chain broke.', \\\"A defendant was shopping in a grocery store when he noticed a woman's purse in the shopping cart unattended. He quickly took the purse and left the store.\\\", 'A defendant broke into a house and took a stereo system. After he had placed the system in his car and was about to leave, the homeowner came home and saw him. She raced to the car and started to hit the defendant through the open window in an attempt to get her stereo back. The defendant punched her in the nose and drove away with the system.', 'In a dark alley, a defendant approached the victim and said, \\\"Give me the gold ring on your finger or I\\\\'ll shoot you with this gun. \\\" The victim gave him the ring. However, the ring really belonged to someone else, as the victim was just borrowing it for the evening to impress his girlfriend. In addition, the gun that the defendant had in his possession was really a water pistol.', 'A defendant walked into a bank and handed a note to the teller demanding money. The teller gave him the money and he left the bank.', 'A defendant found a lost wallet on the street. He took the money and cards from the wallet and threw it in the trash.', 'A defendant entered a store and asked the cashier to show him a necklace. When the cashier turned around to get the necklace, the defendant took money from the cash register and ran away.', 'A defendant picked the lock on an apartment door. The noise startled the tenant, who had been sleeping. The defendant overpowered the tenant, tied him up, and forced him to disclose where he kept his money. The tenant told the defendant to look in the kitchen cabinet, which he did. The defendant found $120 in cash, took the money, and left the apartment.']\",\"question\":\"In which of the following situations would the defendant most likely be found NOT GUILTY of robbery?\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}