{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"a6233a78-e293-5dff-a68c-65888924541c","task_key":"all--test--a6233a78-e293-5dff-a68c-65888924541c","task_revision_id":"1","upstream_id":"","short_description":"Suppose you want to use fuzzing on a program to try to find memory errors; which…","config":"all","split":"test","body":"{\"choices\":\"['You should not use a grammar-based fuzzer, because its adherence to the grammar means it will not find memory errors', 'Compiling the program with address sanitizer (ASAN) will make errors harder to reproduce', 'Compiling the program with address sanitizer (ASAN) will make the source of a memory error easier to find', \\\"Fuzzing doesn't find memory errors, it finds crashes and hangs\\\"]\",\"question\":\"Suppose you want to use fuzzing on a program to try to find memory errors; which of the following statements is true?\",\"subject\":\"computer_security\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}