{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"b8b17a1a-ec2c-53dc-a52e-cc4fb05a0a97","task_key":"all--test--b8b17a1a-ec2c-53dc-a52e-cc4fb05a0a97","task_revision_id":"1","upstream_id":"","short_description":"On May 2, a woman mailed the following letter to a man:\"May 1I have two tickets…","config":"all","split":"test","body":"{\"choices\":\"['Yes, because the letter was mailed on May 2.', 'Yes, because the letter was dated May 1.', 'No, because the man received the letter on May 4.', 'No, because the man did not read the letter until May 6.']\",\"question\":\"On May 2, a woman mailed the following letter to a man:\\\"May 1I have two tickets to the concert on July 1 at the auditorium in town. I'll sell them to you for $60 per ticket, which is $10 above face value. Since the concert has been sold out for months, I think that is a good deal. You have 15 days from the above date to decide whether to accept this offer. \\\"The man received the letter on May 4, but did not read it until May 6. On May 18, the man went to the woman's home and attempted to accept the offer. The woman replied:\\\"Too late! I sold the tickets last week for $75 each. \\\"Assume that the woman's letter created in the man a valid power of acceptance. Was that power terminated by lapse of time before the man went to the woman's home on May 17?\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}