{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"62d2ac4e-2bb8-58ea-bc23-f39d9f6beafc","task_key":"default--test--1163","task_revision_id":"1","upstream_id":"1163","short_description":"A mechanic agreed in writing to make repairs to a landscaper's truck for…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['No, because the mechanic breached the original contract by demanding extra payment.', 'Yes, because the landscaper could have obtained possession of the truck through legal action rather than by agreeing to the increased payment.', \\\"No, because the mechanic exploited the landscaper's immediate need for the truck.\\\", \\\"No, because the landscaper had no reasonable alternative but to yield to the mechanic's wrongful threat.\\\", 'Yes, because the final price is equivalent to the customary charge for such work.', 'No, because the original contract did not include the additional $2,000 payment.', 'Yes, because the landscaper agreed in writing to pay the additional cost.', 'Yes, because the mechanic completed the repairs as agreed.', 'Yes, because the modified contract price did not exceed a reasonable price.', 'No, because the mechanic exerted undue influence over the landscaper with respect to the modification.']\",\"question\":\"A mechanic agreed in writing to make repairs to a landscaper's truck for $12,000. The mechanic properly made the repairs, but when the landscaper tendered payment, the mechanic refused to deliver the truck unless the landscaper promised to pay an additional $2,000. The customary charge for such work was $14,000. Because the landscaper needed the truck immediately to fulfill existing contractual obligations, and because no rental trucks of the same type were available, the landscaper promised in writing to pay the mechanic an additional $2,000. The mechanic then delivered the truck. Will the mechanic be able to enforce the landscaper's promise to pay the additional $2,000?\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}