{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"5c3ef200-19f9-5185-a62b-f4ac97684610","task_key":"default--test--1429","task_revision_id":"1","upstream_id":"1429","short_description":"An engineer signed a two-year contract to serve as the chief safety engineer for…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"[\\\"No, because the risk to the engineer's health excused his nonperformance of the contract.\\\", 'Yes, because the coal mine suffered a financial loss due to the higher salary of the replacement.', 'No, because an at-will employee has the right to terminate an employment contract.', 'Yes, because the mine is reasonably safe for most people.', \\\"Yes, because the coal mine had no knowledge of the engineer's allergy.\\\", 'No, because the engineer was not aware of his allergy at the time of contract signing.', 'No, because the engineer had a valid reason for quitting the job.', 'Yes, because the engineer failed to fulfill the contract obligation.', 'No, because the presence of the chemical agent made the work environment unsafe for the engineer.', \\\"Yes, because the coal mine acted in good faith in mitigating the effect of the engineer's failure to finish the contract term.\\\"]\",\"question\":\"An engineer signed a two-year contract to serve as the chief safety engineer for a coal mine at a salary of $7,000 per month. The position required the engineer to work underground each workday. After one week on the job, the engineer became very ill. He requested testing of the mine air system, which revealed the presence of a chemical agent to which the engineer had a rare allergic reaction. The engineer promptly quit. The coal mine then hired a qualified replacement for the remainder of the engineer's contract at a salary of $7,500 per month. Assume that no statute or regulation applies. If the coal mine sues the engineer for breach of contract, is the coal mine likely to recover damages?\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}