{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"12490817-c670-5029-a581-0a076c1619c5","task_key":"default--test--893","task_revision_id":"1","upstream_id":"893","short_description":"A defendant met her friend at the electronics store where the friend worked.…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['larceny.', 'possession of stolen property.', 'burglary.', 'receiving stolen property.', 'theft.', 'robbery.', 'neither receiving stolen property nor larceny.', 'receiving stolen property and larceny.', 'conspiracy to commit theft.', 'aiding and abetting theft.']\",\"question\":\"A defendant met her friend at the electronics store where the friend worked. Unbeknownst to the defendant, her friend had taken a music player from the display case and stuffed it in her bag earlier in the day. Knowing that employees are subject to search when they leave the store, the friend surreptitiously put the music player into the defendant's bag. The defendant and her friend proceeded to a local tavern and drank shots of tequila. The friend became intoxicated and forgot to retrieve the stolen music player from the defendant's bag. After leaving the tavern, the defendant proceeded home. Upon opening her bag, she was surprised to see the music player. Believing that the music player had accidentally fallen into her bag at the bar, the defendant planned to return it the next day. The following day, however, the defendant changed her mind and decided to keep it. The defendant is guilty of\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}