{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"b45a5c36-fd1b-5c16-8e97-b301d551934c","task_key":"all--test--b45a5c36-fd1b-5c16-8e97-b301d551934c","task_revision_id":"1","upstream_id":"","short_description":"A judge attended a local men's club meeting. In introducing the judge to the…","config":"all","split":"test","body":"{\"choices\":\"['the judge has a reputation for dishonesty in the community.', 'the judge was convicted of bribery three years earlier.', 'the judge, though not convicted, embezzled money from a former employer before being elected to the bench.', 'the judge was convicted for a felony assault 12 years earlier.']\",\"question\":\"A judge attended a local men's club meeting. In introducing the judge to the club members, the club president said: \\\"Gentlemen, I take great pleasure in presenting the judge, who we all know is the best judge money can buy in the city. \\\"The judge sued the club president for slander. If the club president pleads truth as a defense, he may not properly introduce evidence to show\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}