{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"af57b510-ecf5-5e06-9a1c-a80904b11fb7","task_key":"all--test--af57b510-ecf5-5e06-9a1c-a80904b11fb7","task_revision_id":"1","upstream_id":"","short_description":"Ray and Dorothy can both cook and can both pull weeds in the garden on a…","config":"all","split":"test","body":"{\"choices\":\"['Ray pulls weeds, since he has absolute advantage in cooking.', 'Dorothy pulls weeds, since she has absolute advantage in cooking.', 'Dorothy cooks, since she has comparative advantage in cooking.', 'Ray cooks, since he has comparative advantage in cooking.']\",\"question\":\"Ray and Dorothy can both cook and can both pull weeds in the garden on a Saturday afternoon. For every hour of cooking, Ray can pull 50 weeds and Dorothy can pull 100 weeds. Based on this information,\",\"subject\":\"high_school_microeconomics\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}