{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"17c04dce-ce2a-54b5-ad16-82086aa08a98","task_key":"default--test--1259","task_revision_id":"1","upstream_id":"1259","short_description":"A defendant was going out to dinner, and she parked her car at a parking lot…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"[\\\"not guilty, because she did not take possession of another person's property.\\\", 'guilty, because she intentionally avoided paying the parking fee by using her spare keys.', 'guilty, because she acted with the intent to deprive the parking lot of its rightful fee.', 'guilty, because she was under a binding contractual obligation to pay the parking fee.', 'not guilty, because the parking attendant did not have a legal right to withhold her car.', \\\"not guilty, because the parking lot's charges were not agreed upon in advance.\\\", 'not guilty, because the $14 charge was excessively high.', 'guilty, because she failed to pay the $14 before regaining possession of her car.', 'guilty, because she knowingly took her car without the consent of the parking attendant.', 'not guilty, because the defendant cannot be charged with larceny of her own motor vehicle.']\",\"question\":\"A defendant was going out to dinner, and she parked her car at a parking lot across the street from the restaurant. The parking attendant directed the defendant to leave her keys in the car, and he handed her a receipt as she left. Following dinner, the defendant went back to the parking lot and handed her receipt to the attendant who informed her that the parking charge would be $14. Visibly irate, she told the attendant that $14 was much too high for a two-hour parking fee. The parking attendant responded that unless she paid the $14, he could not return her car. She adamantly refused to pay the $14, and her friend, with whom she was having dinner, drove the defendant home in her car. Later that same night, the defendant decided to regain possession of her car with a spare set of keys. She returned to the lot and drove her car home. If the defendant is subsequently charged with larceny, she will most likely be found\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}