{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"bd367357-ad8d-5507-b06f-ae2ec64c2d51","task_key":"default--test--1907","task_revision_id":"1","upstream_id":"1907","short_description":"The local high school football team won the state championship. After the game…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['not succeed, because the quarterback was not responsible for the theft.', 'not succeed, because the quarterback had no control over the actions of the thief.', 'succeed, because the quarterback had an obligation to protect the car.', 'succeed, because the quarterback left the car in his driveway, and it was stolen.', 'not succeed, because the car was returned undamaged.', 'not succeed, because the criminal act of the thief was unforeseeable.', 'succeed, because the quarterback could have returned the car and failed to do so.', \\\"succeed, because the quarterback failed to follow the neighbor's condition to return it by 2:00 a.m.\\\", 'not succeed, because the quarterback intended to return the identical property he borrowed in an undamaged condition.']\",\"question\":\"The local high school football team won the state championship. After the game the high school held a victory rally at the high school gym. As the team was being honored, the star quarterback of the football team approached one of the cheerleaders and asked her out for a date. The cheerleader, who had a crush on the quarterback, said she'd love to go out with him. The quarterback told her, \\\"Great, I'll pick you up at eight. \\\"Later that evening, the quarterback was getting ready to drive to the cheerleader's home when his car wouldn't start. The quarterback then called his next-door neighbor and asked him if he could borrow his car. The neighbor agreed, but told him, \\\"Okay, but make sure you return it by 2:00 a. m. \\\" The quarterback assented and then drove to the cheerleader's house. After picking her up, they drove to an all-night coffee shop where they spent the night talking and catching up over coffee and waffles. Losing track of time, the quarterback and the cheerleader did not leave the coffee shop until 3:00 a. m. After returning home around 4:00 a. m. , the quarterback decided that it was too late to return the car, so he parked it in his driveway. He intended to return the car to the neighbor in the morning. A short while later, a thief stole the neighbor's car from outside the quarterback's home. The police found the car three months later, undamaged. The neighbor, however, refused to accept the car and brought a claim against the quarterback for conversion. In his claim, the neighbor will\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}