{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"6a7e2bf4-29be-5faf-a588-c2a5a7e01e9c","task_key":"all--test--6a7e2bf4-29be-5faf-a588-c2a5a7e01e9c","task_revision_id":"1","upstream_id":"","short_description":"Suppose the market for roses is currently in equilibrium. If the supply of roses…","config":"all","split":"test","body":"{\"choices\":\"['Price and quantity both rise.', 'Price rises, but the change in quantity is ambiguous.', 'Price and quantity both fall.', 'Quantity rises, but the change in price is ambiguous.']\",\"question\":\"Suppose the market for roses is currently in equilibrium. If the supply of roses falls, while at the same time the demand for roses rises, what can you say about the price and quantity of roses in the market?\",\"subject\":\"high_school_microeconomics\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}