{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"d5070eb2-a375-51d0-8d8c-efb782246804","task_key":"default--test--1744","task_revision_id":"1","upstream_id":"1744","short_description":"A man needed to have the oil changed on his car. On Friday, he decided to take…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['lose, because the neighbor had a pre-existing duty to change the oil in the car for $50.', 'win, because the second contract for $75 superseded the original $50 contract.', 'win, because the man willingly offered an additional $25 for the service.', 'win, because the man breached the verbal contract they had.', 'win, because she performed her part of the bargain.', 'win, because the man promised not to sue in small claims court.', 'lose, because the neighbor did not fulfill the initial agreement within the agreed timeframe.', \\\"lose, because the man's offer of an additional $25 was not legally binding.\\\"]\",\"question\":\"A man needed to have the oil changed on his car. On Friday, he decided to take his car to the local dealership to have the oil changed and asked his neighbor if she would give him a ride home from the dealership. The neighbor said, \\\"Why pay the high prices a dealership will charge you? I can change the oil in your car for you. If you will agree to pay me $50, I'll change the oil in your car over the weekend. \\\" The man readily agreed. On Sunday afternoon, the man noticed that his neighbor still had not started working on the car. He asked his neighbor if the car would be ready for him to drive to work Monday morning. The neighbor replied, \\\"I thought about it and realized $50 is too low a price for the work involved. I don't think I'm going to change the oil in your car. \\\" The man then said, \\\"Look, I realize $50 is low for the work involved. If you can change the oil in my car by tomorrow morning, I'll pay you an additional $25. And I won't sue you in small claims court for your failure to perform your promise. \\\" The neighbor then changed the oil late Sunday afternoon, but the man refused to pay to the neighbor anything more than $50. In a suit by the neighbor to recover the additional $25 promised by the man, the neighbor will\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}