{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"cf44dd50-397b-50e4-a4ff-76022b2596fb","task_key":"all--test--cf44dd50-397b-50e4-a4ff-76022b2596fb","task_revision_id":"1","upstream_id":"","short_description":"You are the vice president of public relations for a corporation that produces a…","config":"all","split":"test","body":"{\"choices\":\"['Quickly investigate to make certain your product is definitely the cause of the illness.', 'Stop all sales of the product throughout the nation and issue a recall for that product.', 'Stop sales of the product in the Albany, New York, market and issue a recall for that product.', 'Issue a press release to the Albany, New York, market explaining how to treat the symptoms.']\",\"question\":\"You are the vice president of public relations for a corporation that produces a well-known brand of food products. In the Albany, New York, market, one of your products has recently been found to have some contamination. While apparently not fatal, it has given a large number of consumers severe stomach cramps and other intestinal problems. The cause has been traced back to your product, which is sold throughout the nation. Your CEO wants to know what you would advise to keep the situation from becoming a public relations disaster. What should you recommend?\",\"subject\":\"public_relations\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}