{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"65785d71-9c7c-5d78-a7b7-6dd74c839b55","task_key":"default--test--1167","task_revision_id":"1","upstream_id":"1167","short_description":"A man is at home in his apartment, alone, late at night. Someone repeatedly…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['Robbery and burglary.', 'Robbery, burglary and attempted murder.', 'Burglary, robbery, and trespassing.', 'Aggravated assault and robbery.', 'Attempted murder and trespassing.', 'Burglary and attempted murder.', 'Robbery and attempted murder.', 'Robbery and assault.', 'Trespassing and armed robbery.']\",\"question\":\"A man is at home in his apartment, alone, late at night. Someone repeatedly buzzes his apartment ringer outside the building, but he doesn't answer it because he is very tired and wants to sleep. Later he hears the door to his apartment being kicked in and he is immediately accosted by a man with a gun, who waves the gun at him and says, \\\"You are getting me your cash or else.\\\" The man opens his safe and turns over the money. The perpetrator absconds with the money but is picked up by the police a few minutes later. What common law crimes has the perpetrator probably committed?\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}