{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"0c67ee87-1d58-5978-b3d2-b0e5050091e5","task_key":"default--test--1434","task_revision_id":"1","upstream_id":"1434","short_description":"On May 1, a homeowner and a painter entered into a contract whereby the painter…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['The assignment would constitute a novation, relieving the homeowner of liability.', 'The assignment would be valid only if the homeowner consents to it.', 'The assignment would be effective only after the painting is completed.', 'The assignment would be revocable even if it were reduced to writing and signed by the painter.', 'The assignment would be irrevocable if it were reduced to writing and signed by the painter.', 'The assignment would not constitute a novation, relieving the homeowner of liability.', 'Since personal service contracts of this nature are nonassignable, the homeowner would be under no duty to pay the nephew.', 'The homeowner would not be obligated to pay the nephew, since the contract was nonassignable.', 'The homeowner would be obligated to pay the nephew, since the contract was assignable.']\",\"question\":\"On May 1, a homeowner and a painter entered into a contract whereby the painter was to paint the homeowner's home in exchange for $10,000. The contract stated:\\\"The painting is to be done during the month of June, and payment is to be made within one week after the painting is completed. The rights under this contact are not to be assigned. \\\"The painter started the painting on June 1. On June 10, the painter assigned to his nephew the right to receive payment on the painter's contract with the homeowner. Which of the following statements is most accurate with regard to the painter's assignment to his nephew?\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}