{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"4e4333bd-b5a5-5219-875f-6df10ce851e7","task_key":"default--test--1855","task_revision_id":"1","upstream_id":"1855","short_description":"A wealthy retiree had amassed a fortune after selling his pharmaceutical…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['robbery and murder.', 'murder.', 'robbery and burglary.', 'robbery.', 'burglary and involuntary manslaughter.', 'burglary.', 'aggravated assault and robbery.', 'burglary, robbery, and murder.', 'robbery, burglary, and involuntary manslaughter.', 'involuntary manslaughter.']\",\"question\":\"A wealthy retiree had amassed a fortune after selling his pharmaceutical company. The retiree and his wife lived in an exquisitely decorated penthouse apartment overlooking a park. One night, the retiree and his wife were returning home from dinner when they were accosted by a man outside their apartment building. The man brandished a gun and told the retiree and his wife to take him to their apartment. Upon entering the apartment, the man blindfolded the retiree and his wife and tied them up with duct tape to chairs in the living room. The man then stole the retiree's wallet and assorted jewelry from the bedroom. After the man exited the apartment, the retiree and his wife tried desperately to free themselves from their constraints. As the retiree was struggling, he suffered a heart attack and died. The man should be found guilty of \",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}