{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"aca0d176-5a90-5486-96a0-d42b38599318","task_key":"default--test--1633","task_revision_id":"1","upstream_id":"1633","short_description":"Third-year student is studying contracts. He has learned that generally all…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"[\\\"An assignment that substantially changes obligor's duty or risk.\\\", 'An assignment that changes the nature of the contract.', \\\"An assignment that continues the obligor's duty or risk.\\\", 'An assignment that transfers rights to an incapable party.', 'An assignment that causes undue hardship or loss to the obligor.', 'An assignment that reduces the value of the contract to the obligor.', 'An assignment of future rights from future contracts.', 'An assignment that is made without the consent of the other party.', 'An assignment not allowed by law.']\",\"question\":\"Third-year student is studying contracts. He has learned that generally all contractual rights may be assigned, but there are some exceptions. Which of the following is not an exception?\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}