{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"751d853e-aa0c-5d0a-984d-17997496bec6","task_key":"all--test--751d853e-aa0c-5d0a-984d-17997496bec6","task_revision_id":"1","upstream_id":"","short_description":"A 73-year-old woman comes to the physician because of a 2-month history of…","config":"all","split":"test","body":"{\"choices\":\"['Vitamin B1 (thiamine)', 'Vitamin B2 (riboflavin)', 'Vitamin B6 (pyridoxine)', 'Vitamin B12 (cyanocobalamin)']\",\"question\":\"A 73-year-old woman comes to the physician because of a 2-month history of diffuse weakness and tingling of her arms and legs. Neurologic examination shows weakness of the extensor and flexor muscles of the lower extremities. Knee and ankle deep tendon reflexes are exaggerated. Sensation to vibration and position is decreased in all extremities, but the decrease is more prominent in the lower extremities than in the upper extremities. This patient most likely has a deficiency of which of the following vitamins?\",\"subject\":\"professional_medicine\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}