{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"a11dbeee-2540-5579-909e-597e99831231","task_key":"default--test--927","task_revision_id":"1","upstream_id":"927","short_description":"On February 15, a company that manufactures metal sidings for home exteriors…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['Yes, because the company did not notify the builder that the 1/2-inch sheets were for accommodation only.', \\\"No, because the company could accept the builder's offer by prompt shipment of either conforming or nonconforming goods.\\\", 'No, because the builder accepted the goods by not returning them immediately.', 'Yes, because the company failed to deliver by the stipulated April 1 deadline.', 'Yes, because the aluminum sheets were nonconforming goods.', 'No, because the builder waived its right to reject the nonconforming goods by not returning them promptly to the company.', 'No, because the company has the right to ship nonconforming goods as long as they meet the overall order requirements.', 'No, because the builder did not specify the reason for rejection in the fax.']\",\"question\":\"On February 15, a company that manufactures metal sidings for home exteriors received the following order from a builder: \\\"Please ship 300 sheets of 1/4-inch refabricated aluminum siding. Delivery by April 1. \\\"On March 8, the company shipped 300 sheets of 1/2-inch refabricated aluminum siding, which were received by the builder on March 10. The following day, the builder sent the following fax to the company: \\\"Be advised that your shipment is rejected. Order stipulated 1/4-inch sheets. \\\" This fax was received by the company, but the builder did not ship the nonconforming aluminum sheets back to the company. Did the builder properly reject the shipment delivered on March 10?\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}