{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"64db773f-2690-51f6-a2a3-d621d325f340","task_key":"default--test--945","task_revision_id":"1","upstream_id":"945","short_description":"Under which one of the following situations would the defendant(s) most likely…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['A defendant, a trusted employee, was given the keys to the safe at the business where he worked. One night, he took money from the safe, intending to return it after he won at the horse races.', 'A defendant found an envelope full of money on a park bench. Knowing it was probably lost property, he decided to keep it instead of turning it in to the police.', 'A defendant who was a contractor took payment from a homeowner for materials and then used those materials on a different job.', 'During a craps game in the back room of a bar, the defendant lost $150 to the victim. The defendant left the bar after losing his money and returned an hour later with a gun in his possession. Honestly believing that the $150 still belonged to him, the defendant confronted the victim in the back room and demanded the return of his money. Frightened, the victim handed the money back to the defendant.', 'As two defendants were walking down the street, they noticed a victim park his metallic gold sports car and enter the pool hail. When they approached the car, one of the defendants observed that the victim had left the keys in the ignition. Knowing that the victim would be hustling pool all evening, the defendants hopped into the sports car and drove off, intending to return the car later that evening.', \\\"One afternoon, the defendant noticed the victim riding his racing bike in the park. The defendant, who always wanted to own a racing bike, saw his opportunity when the victim left his bike unattended to participate in a touch football game. The defendant jumped on the bike and quickly rode away. Later that evening, the defendant called the victim and demanded $200 for the return of the bike. The victim agreed to the defendant's demand. The following day, the victim paid the defendant the money, and the defendant, in turn, returned the bike.\\\"]\",\"question\":\"Under which one of the following situations would the defendant(s) most likely be found guilty of larceny?\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}