{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"e9a469ef-d341-5e4a-8644-f4239fefb8f5","task_key":"default--test--1912","task_revision_id":"1","upstream_id":"1912","short_description":"A toy collector had purchased 10 antique toys over the last several years and…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"[\\\"Yes, because the expert sent the June 7 note before learning of the collector's death.\\\", 'Yes, because the previous successful contracts set a precedent for this one.', 'No, because the collector died before the expert sent the June 7 note.', 'No, because the contract was not formally written and signed.', \\\"No, because the expert was not aware of the collector's death before starting the restoration.\\\", 'No, because the offer lapsed when the collector died.', 'Yes, because the expert had already accepted the work and payment terms.', \\\"Yes, because the offer was accepted before the collector's death.\\\", \\\"No, because the collector's estate did not confirm the contract.\\\"]\",\"question\":\"A toy collector had purchased 10 antique toys over the last several years and had had them restored by an expert in toy restoration. On June 1, the collector sent the 11th antique toy to the expert with a signed note that read: \\\"Here is another toy for you to restore. As with all prior jobs, I will pay $500 for the work, but no more.\\\" On June 4, after receipt of the collector's June 1 note and the toy, the expert began restoring the toy. On June 6, the collector unexpectedly died. On June 7, unaware of the collector's death, the expert sent the collector a note that stated that the restoration work had begun on June 4. The following day, the expert learned of the collector's death. Does a contract exist that binds the expert and the collector's estate?\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}