{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"93433ff2-9c28-54a1-8bde-61f9722ebe89","task_key":"all--test--93433ff2-9c28-54a1-8bde-61f9722ebe89","task_revision_id":"1","upstream_id":"","short_description":"A retailer that sells footwear maintains a single database containing records…","config":"all","split":"test","body":"{\"choices\":\"['Which items listed in the database are not currently in the store', 'Which colors are more popular among men than women', 'Which type of footwear is most popular among adults', 'The total number of shoes sold in a particular month']\",\"question\":\"A retailer that sells footwear maintains a single database containing records with the following information about each item for sale in the retailer's store.\\n\\n   ° Item identification number\\n   ° Footwear type (sneakers, boots, sandals, etc.)\\n   ° Selling price (in dollars)\\n   ° Size\\n   ° Color\\n   ° Quantity available\\n\\n Using only the database, which of the following can be determined?\",\"subject\":\"high_school_computer_science\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}