{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"a187bc24-95da-5cd2-a146-575b982f5f34","task_key":"default--test--5639","task_revision_id":"1","upstream_id":"5639","short_description":"Which of the following transactions correctly illustrates the doctrine of…","config":"default","split":"test","body":"{\"category\":\"other\",\"options\":\"[\\\"A contract required hair styling to be done to Toby's satisfaction but Toby was in good faith dissatisfied with the completed result.\\\", 'Leslie painted an entire room but failed to put the electrical outlet covers back on.', 'Blair ordered a dozen blue chairs from Kyle but Kyle delivered a dozen red chairs.', 'A carpenter built a bookshelf but used pine instead of the contracted oak.', \\\"A dentist competently completed the extraction of Lee's tooth but mistakenly pulled the wrong one.\\\", 'A builder constructed a house but forgot to install the kitchen sink.', 'A chef cooked a meal but used pepper instead of salt.', 'A mechanic fixed a car but forgot to change the oil.']\",\"question\":\"Which of the following transactions correctly illustrates the doctrine of substantial performance?\",\"src\":\"ori_mmlu-professional_accounting\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}