{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"6bef9167-9d58-5a29-83d8-0259b0c9a553","task_key":"all--test--6bef9167-9d58-5a29-83d8-0259b0c9a553","task_revision_id":"1","upstream_id":"","short_description":"Avalon had a wild time during spring break. Unfortunately the guy she hooked up…","config":"all","split":"test","body":"{\"choices\":\"['Avalon could not contract genital herpes from performing fellatio (oral-genital sex) on the guy.', 'If Avalon contracted genital herpes she can be cured with antibiotics if administered immediately after the first occurrence.', 'If Avalon contracted genital herpes she will only be contagious when she has a break out.', 'none of the above']\",\"question\":\"Avalon had a wild time during spring break. Unfortunately the guy she hooked up with in Palm Beach had genital herpes. Which of the following is true about genital herpes?\",\"subject\":\"human_sexuality\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}