{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"7046fccb-57ab-527f-a451-984d610a29f2","task_key":"all--test--7046fccb-57ab-527f-a451-984d610a29f2","task_revision_id":"1","upstream_id":"","short_description":"An aspiring actress, whose auditions had not been going well, was forced to pawn…","config":"all","split":"test","body":"{\"choices\":\"['guilty of larceny, but not conspiracy to commit larceny.', 'guilty of conspiracy to commit larceny, but not larceny.', 'guilty of both larceny and conspiracy to commit larceny.', 'not guilty of either larceny or conspiracy to commit larceny.']\",\"question\":\"An aspiring actress, whose auditions had not been going well, was forced to pawn her mother's pearls to pay rent and buy food. She finally got good news when her agent told her that she had landed a part in a big movie. Delighted, the actress went down to the pawnshop to retrieve her mother's pearls. When she arrived, the clerk told her that she needed to have cash in order to retrieve the necklace. As she had not been paid yet for the movie, the actress called her agent and told him her dilemma. The agent told her he would come down to the pawnshop and fix everything. When the agent arrived, the actress was still arguing with the clerk. The agent pulled the actress aside and told her that he would distract the clerk while the actress snatched the necklace. The agent then pretended to be interested in an antique lighter, and the actress snatched her necklace off the counter and left the store. The agent is subsequently prosecuted for larceny. The defendant could most appropriately be found\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}