{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"5f6f06bd-55e6-5259-99c8-0d2897e75a2d","task_key":"default--test--1775","task_revision_id":"1","upstream_id":"1775","short_description":"On May 2, a woman mailed the following letter to a man:\"May 1I have two tickets…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['Yes, because the man did not read the letter until after the 15-day period.', 'No, because the woman sold the tickets before the man could accept the offer.', 'Yes, because the letter was mailed on May 2.', 'No, because the man attempted to accept the offer within 15 days of receiving the letter.', 'Yes, because the letter was dated May 1.', 'No, because the man did not read the letter until May 6.', \\\"No, because the man was still within the 15-day period when he went to the woman's home.\\\", 'No, because the man received the letter on May 4.', 'Yes, because the woman had already sold the tickets by the time the man went to her home.', 'Yes, because the man attempted to accept the offer after 15 days from the date of the letter.']\",\"question\":\"On May 2, a woman mailed the following letter to a man:\\\"May 1I have two tickets to the concert on July 1 at the auditorium in town. I'll sell them to you for $60 per ticket, which is $10 above face value. Since the concert has been sold out for months, I think that is a good deal. You have 15 days from the above date to decide whether to accept this offer. \\\"The man received the letter on May 4, but did not read it until May 6. On May 18, the man went to the woman's home and attempted to accept the offer. The woman replied:\\\"Too late! I sold the tickets last week for $75 each. \\\"Assume that the woman's letter created in the man a valid power of acceptance. Was that power terminated by lapse of time before the man went to the woman's home on May 17?\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}