{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"b5eb7023-511a-503f-8495-3b85f39e2580","task_key":"all--test--b5eb7023-511a-503f-8495-3b85f39e2580","task_revision_id":"1","upstream_id":"","short_description":"A sports talk show asks people to call in and give their opinion of the…","config":"all","split":"test","body":"{\"choices\":\"[\\\"They will most likely feel that the officiating could have been better, but that it was the team's poor play, not the officiating, that was primarily responsible for the loss.\\\", 'They would most likely call for the team to get some new players to replace the current ones.', \\\"The team probably wouldn't have lost if the officials had been doing their job.\\\", 'Because the team had been foul-plagued all year, the callers would most likely support the officials.']\",\"question\":\"A sports talk show asks people to call in and give their opinion of the officiating in the local basketball team's most recent loss. What will most likely be the typical reaction?\",\"subject\":\"high_school_statistics\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}