{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"23e439fc-c443-5fad-9799-7c2c78e720eb","task_key":"all--test--23e439fc-c443-5fad-9799-7c2c78e720eb","task_revision_id":"1","upstream_id":"","short_description":"The mean height of adult men is 70 inches, with a standard deviation of four…","config":"all","split":"test","body":"{\"choices\":\"['It cannot be determined, because the mean heights for the two groups are different.', 'It cannot be determined, because the standard deviations of the heights for the two groups are different.', 'The man, because he is 74 inches tall and the woman is only 70 inches tall.', 'The woman, because her height is 1.33 standard deviations above the mean height of all women, whereas the man’s height is only one standard deviation above the mean height of all men.']\",\"question\":\"The mean height of adult men is 70 inches, with a standard deviation of four inches. The mean height of adult women is 66 inches, with a standard deviation of three inches. Between a man with a height of 74 inches and a woman with a height of 70 inches, who is more unusually tall within his or her respective sex?\",\"subject\":\"high_school_statistics\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}