{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"9800c556-2fff-5b74-96ac-83ff77132f43","task_key":"default--test--1441","task_revision_id":"1","upstream_id":"1441","short_description":"A woman needed to have her microwave repaired. She contacted the local handyman,…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['The woman has a cause of action against the repair store, but only if the handyman does not pay the repair bill.', 'The woman has a cause of action against the handyman and the repair store, but only if the repair is not completed on time.', 'The woman has no cause of action against either the repair store or the handyman.', 'The woman has no cause of action as she agreed to the repair without a specific completion date.', 'The woman has a cause of action against the repair store, but only if they knew she was the true owner of the microwave.', 'The woman has a cause of action against the repair store only.', 'The woman has a cause of action against the handyman, but only if the microwave is not repaired.', 'The woman has a cause of action against the handyman only.', 'The woman has a cause of action against both the repair store and the handyman.', 'The woman has a cause of action against both the repair store and the handyman, but only if the handyman fails to pay the repair bill.']\",\"question\":\"A woman needed to have her microwave repaired. She contacted the local handyman, who said he could repair the microwave for $100. The woman readily agreed and delivered the microwave to the handyman, who promised to have it ready in two weeks. One week later, the handyman realized that he had so much work to do that he would not be able to repair the microwave on time. He then took the microwave to a repair store. The repair store agreed to repair the microwave for $80 within one week. The owner of the repair store that contracted with the handyman was unaware that the woman actually owned the microwave. If the repair store fails to repair the microwave, which of the following is the most accurate statement?\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}