{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"a988a9ef-5418-5780-9942-f0cb9692a57a","task_key":"default--test--1263","task_revision_id":"1","upstream_id":"1263","short_description":"A defendant and a classmate were students at a state college. They were spending…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['responsibility as the owner of the vehicle involved in the accident.', 'recklessness in lending his car to his classmate.', 'liability for providing alcohol to his classmate.', 'participation in the drinking session prior to the accident.', 'indirect involvement in the accident due to his car being involved.', 'joint venture in lending his car to his classmate for a common purpose.', 'negligence in not maintaining the car’s brake system.', 'responsibility for the accident as an accomplice.', 'negligence in not preventing his classmate from driving drunk.', 'vicarious liability for the conduct of his classmate.']\",\"question\":\"A defendant and a classmate were students at a state college. They were spending a leisurely afternoon listening to music in the defendant's dorm room. They had just consumed a six-pack of beer when the classmate asked the defendant if there was anything else to drink. The defendant indicated there wasn't and suggested that his classmate drive to the store and buy another six-pack. The classmate told the defendant that his car wasn't working and asked the defendant if he could borrow his. The defendant assented and gave his classmate the keys to his car. The defendant knew that the classmate was drunk when he lent him his car. The classmate was driving to the store at an excessive rate of speed. As he approached an intersection, he was traveling at 70 M. PJ-I. When he came to the intersection, the light turned red. The classmate, who made no effort to stop in time, drove through the red light and collided with another car. The driver of the other car, who had entered the intersection with a green light, was killed in the accident. In the event that the defendant can be convicted of manslaughter, it will most likely be upon the basis of\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}