{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"89c9ef06-47c7-54c0-89b7-7a733a952777","task_key":"default--test--1800","task_revision_id":"1","upstream_id":"1800","short_description":"According to usual practice, a long-time regular customer left her car at an…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['No, because there is always an implication that the provider must get permission before doing work not approved.', 'No, because there was no written agreement specifying the cost of the services.', 'Yes, because she left written instructions, which can be considered a contract.', 'Yes, because she had a history of leaving her car for service without discussing price or details.', 'Yes, because the circumstances, including prior practice, showed intent to contract and a meeting of the minds.', 'No, because a contract cannot be based on implications and guess work.', 'Yes, because the circumstances and past practice established an implied-in-law contract based on mutual assent.', 'No, because the mechanic did not provide an estimate before performing the services.', 'No, because she did not explicitly agree to the price of the services.', 'Yes, because the quality of the work was satisfactory and pertained to the areas in the instructions.']\",\"question\":\"According to usual practice, a long-time regular customer left her car at an auto mechanic's shop with written instructions to \\\"check and tune up the engine; make sure the brakes are good and working; examine the tires; also, check and repair all major systems because I am driving cross country next week.\\\" She did not see the mechanic or discuss with him the price or other details. When she picked up the car, there was a bill for $5,000 on the dash board. All of the services and parts were workmanlike, not defective and pertained to the areas in the instructions. She refused to pay, claiming that there was no agreement and that the mechanic should have called for advance approval. Can she be held responsible for contractual services performed?\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}