{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"b960b782-a8e0-5085-956e-3d81a375fd34","task_key":"default--test--1917","task_revision_id":"1","upstream_id":"1917","short_description":"A defendant went to a party at his friend's house. While he was there, the…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['Larceny.', 'Larceny, burglary, and attempted extortion.', 'Robbery, extortion, and assault.', 'Robbery.', 'Robbery and extortion.', 'Larceny and attempted extortion.', 'Extortion.', 'Assault.', 'Burglary.', 'Larceny and burglary.']\",\"question\":\"A defendant went to a party at his friend's house. While he was there, the defendant drank a lot of beers and countless shots. The defendant eventually stumbled into the bathroom where he spotted a purse lying on the floor under the sink. Wondering vaguely if it might contain more beer, he opened it up and found his friend's wallet. He grabbed some money from the wallet and left the bathroom. As he was descending the stairs with the money in his hand, his friend spotted him. She was initially concerned he might tumble down the stairs but upon seeing the money in his hand, she asked him where he got it. The defendant responded, \\\"Ha! Your money or your life!\\\" Not taking him seriously, the friend let him pass by. It wasn't until later that she realized that the defendant had taken the money from her wallet. The defendant should be charged with which of the following crimes?\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}