{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"0b182731-3694-5892-974c-e28795493247","task_key":"default--test--1073","task_revision_id":"1","upstream_id":"1073","short_description":"In which situation would the defendant most likely be guilty of murder?","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"[\\\"The defendant, a surgeon, intentionally makes a mistake during a patient's surgery, which results in the patient's death.\\\", 'The defendant leaves a bottle of poison on his kitchen counter. His toddler son finds the bottle, drinks from it, and dies.', 'As a practical joke, the defendant throws a firecracker in a crowded movie theater. A victim, who has a weak heart, becomes frightened and thinks someone is firing a gun. The victim suffers a heart attack and dies.', \\\"The defendant keeps a loaded gun in his house for protection. One day, the defendant leaves the gun on the kitchen table and leaves for work. That same day, a robber breaks into the defendant's home, sees the gun, and steals it. The robber then uses the gun to rob a convenience store. During the robbery, the robber shoots and kills the proprietor with the defendant's gun.\\\", 'The defendant, knowing that his friend is severely allergic to peanuts, invites the friend over for dinner and serves a meal containing peanuts. The friend has a severe allergic reaction and dies.', 'The defendant hates her husband after learning that he is having an affair with her best friend. She also knows that her husband owes a man a large amount of money, which he refuses to pay. Before the husband is about to leave for work, the defendant looks out the window and sees the man hiding outside with a gun. The defendant decides not to warn the husband, who then walks outside and is shot and killed by the man.', \\\"The defendant allows a friend to borrow his car, knowing that the friend has never driven before and does not have a driver's license. The friend crashes the car into a tree, killing a pedestrian.\\\", 'The defendant, a pharmacist, intentionally dispenses the wrong medication to a patient. The patient takes the medication and dies.', 'The defendant sends a threatening letter to a coworker. The coworker, who has a weak heart, is so frightened by the letter that he suffers a heart attack and dies.', \\\"The defendant and his friend attended a party together. At the party, the friend became extremely inebriated. The friend asked the defendant if she could borrow his car to drive to the liquor store to buy more booze for the party. Although the defendant was aware of the friend's intoxicated condition, he allowed her to drive his car to the liquor store. The friend drove through a red light and collided with the victim's car. The victim was killed in the accident.\\\"]\",\"question\":\"In which situation would the defendant most likely be guilty of murder?\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}