{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"18c80d9d-c3d8-50b6-9149-a4e0aec1e2e3","task_key":"all--test--18c80d9d-c3d8-50b6-9149-a4e0aec1e2e3","task_revision_id":"1","upstream_id":"","short_description":"The owners of independent gasoline stations in a community meet secretly and…","config":"all","split":"test","body":"{\"choices\":\"['found guilty under the Sherman Antitrust Act and face prison time', 'found guilty under the Magnuson-Moss Warranty Act and face large fines', 'found guilty under the Adamson Act and face prison time and fines', 'found not guilty because retailers are allowed to set their own price levels']\",\"question\":\"The owners of independent gasoline stations in a community meet secretly and agree to keep their prices high to protect their profitability. If their plan is discovered and they are charged with a crime, they could be\",\"subject\":\"marketing\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}