{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"fcc298c2-7b51-5b34-863e-5589b6d29fc7","task_key":"default--test--1517","task_revision_id":"1","upstream_id":"1517","short_description":"A senior associate in a law firm was conducting spot checks of other associates…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['Involuntary manslaughter.', 'Negligent homicide.', 'Voluntary manslaughtçr.', 'Assault.', 'Criminal Threat.', 'No crime, it was self-defense.', 'Attempted murder.', 'Battery.', 'Aggravated assault.', 'Murder.']\",\"question\":\"A senior associate in a law firm was conducting spot checks of other associates as they left the office to determine who had taken some documents that were missing from the firm's library. A young associate was leaving his office for the day when he was stopped by the defendant, who asked him to open his briefcase. The senior associate explained that some files from the law office had been stolen. Startled, the young associate refused to permit the senior associate to look in his briefcase. The young associate said that he had personal papers in his briefcase and he didn't want the senior associate to see them. Having his suspicions aroused, the senior associate raised his clenched fist and exclaimed, \\\"If you don't hand over that briefcase, I'm going to punch you in the mouth. \\\" The young associate, who, unknown to the senior associate, suffered from a serious heart condition, had a seizure and died as a result of fright produced by the threatened attack. Which of the following is the most serious crime for which the senior associate should be found guilty? \",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}