{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"7cc445db-c96a-5c56-bfc9-3fe404be94eb","task_key":"all--test--7cc445db-c96a-5c56-bfc9-3fe404be94eb","task_revision_id":"1","upstream_id":"","short_description":"A proficient engineer can easily design skeletal structures that are more…","config":"all","split":"test","body":"{\"choices\":\"['natural selection has not had sufficient time to create the optimal design in each case, but will do so given enough time.', 'natural selection operates in ways that are beyond the capability of the human mind to comprehend.', 'in many cases, phenotype is not merely determined by genotype, but by the environment as well.', 'natural selection is generally limited to modifying structures that were present in previous generations and in previous species.']\",\"question\":\"A proficient engineer can easily design skeletal structures that are more functional than those currently found in the forelimbs of such diverse mammals as horses, whales, and bats. That the actual forelimbs of these mammals do not seem to be optimally arranged is because\",\"subject\":\"high_school_biology\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}