{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"ae425ff6-e25b-5095-8186-f441812a963f","task_key":"default--test--1195","task_revision_id":"1","upstream_id":"1195","short_description":"A defendant went to a convenience store intending to rob it. The defendant had a…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['guilty, because the defendant entered the store with a gun in his possession.', 'guilty, because the owner was placed in fear.', 'not guilty, because the owner gave permission to take the money.', 'guilty, because the defendant had a weapon.', \\\"not guilty, because the defendant did not take any money from the victim's person.\\\", 'not guilty, because the owner did not see the gun.', 'guilty, because the defendant took money from the store.', 'not guilty, because the owner did not directly interact with the defendant.', 'guilty, because the defendant intended to rob the store.', 'not guilty, because the defendant did not make any threat of force.']\",\"question\":\"A defendant went to a convenience store intending to rob it. The defendant had a gun inside his coat pocket. When the defendant entered the store, the owner saw that he had his hand in his coat pocket. Although the owner did not actually see the gun, he noticed a bulge in the defendant's coat pocket. Paranoid because of a rash of recent robberies, the owner said, \\\"Please don't hurt me 11 do anything you want. \\\" The owner then fainted and fell to the floor. The defendant walked behind the counter and opened the cash register. He took the money from the register and left the store. If the defendant is charged with robbery, he should be found\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}