{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"6738ea43-6fc9-5cbb-803d-357e258a4b85","task_key":"default--test--1710","task_revision_id":"1","upstream_id":"1710","short_description":"A popular singer expressed interest in performing in a theater producer's…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['Restitution for the cost of finding a replacement singer.', 'Expectation damages in the form of the profits that the production would have made if it had been performed.', 'Specific performance requiring the singer to perform in the musical.', 'Damages for the lost ticket sales resulting from the cancellation of the production.', 'Nothing, because it was not reasonable for the producer to expect to be reimbursed for expenses incurred when he knew that the singer was not obligated until there was a final, signed agreement.', 'Reliance damages in the form of the actual expenses the producer incurred while negotiating with the singer.', \\\"Restitution for the damage to the producer's business relationships due to the failure of the production.\\\", 'Reliance damages in the form of the expenses the producer would have had if the production had gone ahead.', \\\"Restitution for any harm done to the producer's reputation when the production was abandoned.\\\", \\\"Punitive damages due to the singer's failure to agree on the contract terms.\\\"]\",\"question\":\"A popular singer expressed interest in performing in a theater producer's musical production. The producer and the singer agreed that there would be no enforceable contractual obligations between them until they had entered into a final, signed contract. They then began negotiating the terms of the contract. While the parties were negotiating, the producer began to spend money on the production. When the parties were unable to agree on the terms of a the parties were unable to agree on the terms of a final contract, the producer made reasonable but unsuccessful efforts to hire another singer before abandoning the production. If the producer sues the singer, what will he likely recover?\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}