{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"f9f85e7a-5269-5825-9013-4ceaed56a000","task_key":"all--test--f9f85e7a-5269-5825-9013-4ceaed56a000","task_revision_id":"1","upstream_id":"","short_description":"Fourth-grader Billy Bully started a fist fight with 10-year-old Jack DeLong.…","config":"all","split":"test","body":"{\"choices\":\"['He thought Jack was the initial aggressor.', 'Billy had a valid defense if Jack sued him.', 'The force Ike used was not excessive.', 'He had no intention of injuring Jack, only to stop his beating of Billy.']\",\"question\":\"Fourth-grader Billy Bully started a fist fight with 10-year-old Jack DeLong. Billy thought Jack was a weakling, but Jack had been working out with weights and knew karate. Ike Intervenor, a large 25-year-old who knew neither party, strolled around the corner of the block in time to see Jack strike Billy twice. Billy fell back and Jack stepped forward to strike him again. Ike thought Jack was the aggressor and to protect Billy, Ike threw Jack into the bushes, accidentally breaking Jack's arm. Jack brought suit against Ike for his damages. Ike's best defense is\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}