{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"250a658a-4191-546a-88ff-47d8e7d96435","task_key":"default--test--1542","task_revision_id":"1","upstream_id":"1542","short_description":"A man and a woman were driving eastbound along a steep two-lane winding highway.…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['the man is guilty of involuntary manslaughter, and the woman is guilty of failing to report an accident.', 'the man is guilty of manslaughter only.', 'the man is guilty of vehicular homicide, and the woman is not guilty.', 'the man and the woman are both guilty of manslaughter.', 'the woman is guilty of manslaughter only.', 'the woman is guilty of accessory to manslaughter.', 'the man is guilty of involuntary manslaughter, and the woman is guilty of accessory after the fact.', 'the man is guilty of reckless driving causing death, and the woman is not guilty.', 'neither the man nor the woman is guilty of manslaughter.']\",\"question\":\"A man and a woman were driving eastbound along a steep two-lane winding highway. The woman was driving in front of the man. As the man approached the rear of the woman's car, he became impatient and tried to pass her along a section of the highway designated as a no-passing zone. When the man swerved his car into the westbound lane to pass the woman, he didn't see another driver, who was rounding a curve in the westbound lane. In order to avoid a head-on collision with the man, the other driver swerved his car to the shoulder of the roadway. The other driver, however, was unable to control his car along the narrow strip of the shoulder, and it fell down the steep mountain. The man and the woman both saw the other driver's car slide down the hillside. They immediately stopped their vehicles and walked over to the edge of the roadway looking for the car. They saw the other driver's car overturned in a gully about 200 feet down the mountain. The man and the woman heard the other driver moaning and calling for help. However, the man and the woman failed to provide any assistance. They walked back to their cars and drove off. Although the man and the woman passed through a business area a short while later, they neither reported the accident nor sought aid for the other driver. Hours later, the other driver died from injuries suffered in the accident. If the man and the woman are subsequently prosecuted for the other driver's death, the most likely outcome would be that\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}