{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"4b1b92eb-1fd0-5567-8ee3-5b736dc3c276","task_key":"all--test--4b1b92eb-1fd0-5567-8ee3-5b736dc3c276","task_revision_id":"1","upstream_id":"","short_description":"This question refers to the following information.","config":"all","split":"test","body":"{\"choices\":\"['Pride in the mastery of the military arts', 'Chivalry', 'Civic pride', 'Patronage of the arts']\",\"question\":\"This question refers to the following information.\\nFlorence is more beautiful and five hundred forty years older than your Venice. … We have round about us thirty thousand estates, owned by nobleman and merchants, citizens and craftsman, yielding us yearly bread and meat, wine and oil, vegetables and cheese, hay and wood, to the value of nine thousand ducats in cash. … We have two trades greater than any four of yours in Venice put together—the trades wool and silk. … Our beautiful Florence contains within the city … two hundred seventy shops belonging to the wool merchant's guild, from whence their wares are sent to Rome and the Marches, Naples and Sicily, Constantinople … and the whole of Turkey. It contains also eighty-three rich and splendid warehouses of the silk merchant's guild.\\nBenedetto Dei, \\\"Letter to a Venetian,\\\" 1472\\nThe passage may be used as evidence for the existence of which of the following Renaissance cultural characteristics?\",\"subject\":\"high_school_european_history\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}