{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"2dc8b99e-2db2-5990-a1a9-b885149462ef","task_key":"default--test--1386","task_revision_id":"1","upstream_id":"1386","short_description":"A defendant was angry at his friend for marrying the defendant's former…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['The gun discharged accidentally.', 'The killing was the result of negligence in missing the friend.', 'He was under the influence of alcohol at the time.', 'The bullet was deflected and did not hit the intended target.', 'He was acting out of anger and not with a deliberate intention.', 'He did not intend to kill anyone, only to scare the friend.', 'He intended to kill the friend and not the daughter.', 'He did not know the daughter was present.', 'He intended only to wound the friend.', 'He was unaware of the elements of malice.']\",\"question\":\"A defendant was angry at his friend for marrying the defendant's former girlfriend. As the friend was painting his house one afternoon, the defendant fired a shot from his gun at him. Although the shot missed the friend, the bullet struck and killed the friend's daughter. The defendant is subsequently charged with the first-degree murder of the daughter. The relevant statutes in effect in this jurisdiction are as follows:Section 169: Murder in the first degree is the unlawful and intentional killing of a human being with malice aforethought. Section 170: Malice is expressed when there is manifested a deliberate intention to take away the life of another. Which of the following, if established, would provide the defendant with his best defense?\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}