{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"faa08602-2d95-5cae-b927-89f32d0f6e0e","task_key":"all--test--faa08602-2d95-5cae-b927-89f32d0f6e0e","task_revision_id":"1","upstream_id":"","short_description":"Buffy and Willow were discussing their sexual fantasies. Buffy revealed that she…","config":"all","split":"test","body":"{\"choices\":\"[\\\"Buffy's fantasies suggest that she must be a latent homosexual or bisexual.\\\", \\\"Buffy's fantasies are indicative of a paraphilia.\\\", \\\"Buffy's fantasies indicate that she is not getting enough sex to keep her satisfied.\\\", \\\"Buffy's fantasies are fairly typical and tell us little about her sexual tendencies.\\\"]\",\"question\":\"Buffy and Willow were discussing their sexual fantasies. Buffy revealed that she sometimes fantasizes about sex with other women even though she normally finds herself attracted to men. She also fantasizes about group sex. Which of the following statements can be inferred from Buffy's fantasies?\",\"subject\":\"human_sexuality\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}