{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"d42b077c-7a34-5c20-9f1a-43bc7f3c1fb0","task_key":"default--test--1197","task_revision_id":"1","upstream_id":"1197","short_description":"After a severe thunderstorm had struck the area, a homeowner discovered that her…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['The contract would be unenforceable, because it was not written and signed by both parties.', 'The performance of the roofer would be a constructive condition precedent to the performance by the homeowner.', 'The performance by the homeowner would be an implied condition subsequent to the performance by the roofer.', 'The performance by the homeowner would be a constructive condition precedent to the performance by the roofer.', 'The performances of the homeowner and the roofer would be constructive concurrent conditions.', 'The contract would be unenforceable, because it was oral.', 'The performance of the roofer would be an implied condition subsequent to the performance by the homeowner.', 'The performances of the homeowner and the roofer would be independent conditions.', 'The performances of the homeowner and the roofer would be express concurrent conditions.']\",\"question\":\"After a severe thunderstorm had struck the area, a homeowner discovered that her roof had suffered extensive damage. She telephoned a roofer to inquire about having the necessary repairs done to her roof. The roofer agreed to finish the work within 30 days, and the homeowner promised to pay$10,000. Which of the following statements is most accurate?\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}