{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"40ec0c38-e856-556e-9910-986472f3bac3","task_key":"default--test--5793","task_revision_id":"1","upstream_id":"5793","short_description":"You are the vice president of public relations for a corporation that produces a…","config":"default","split":"test","body":"{\"category\":\"other\",\"options\":\"['Conduct an internal investigation but do not publicly acknowledge the issue.', 'Issue a press release to the Albany, New York, market explaining how to treat the symptoms.', 'Stop all sales of the product throughout the nation and issue a recall for that product.', 'Quickly investigate to make certain your product is definitely the cause of the illness.', 'Only recall the product in stores where complaints have been received.', 'Continue sales but include warning labels on the product packaging.', 'Offer free product vouchers as compensation to affected consumers.', 'Deny any responsibility until more information is available.', 'Stop sales of the product in the Albany, New York, market and issue a recall for that product.', 'Issue a press release downplaying the severity of the illness.']\",\"question\":\"You are the vice president of public relations for a corporation that produces a well-known brand of food products. In the Albany, New York, market, one of your products has recently been found to have some contamination. While apparently not fatal, it has given a large number of consumers severe stomach cramps and other intestinal problems. The cause has been traced back to your product, which is sold throughout the nation. Your CEO wants to know what you would advise to keep the situation from becoming a public relations disaster. What should you recommend?\",\"src\":\"ori_mmlu-public_relations\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}