{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"3e8c7eb0-bf18-5d83-865f-a144ed702162","task_key":"all--test--3e8c7eb0-bf18-5d83-865f-a144ed702162","task_revision_id":"1","upstream_id":"","short_description":"A store published the following advertisement in a local newspaper on Monday,…","config":"all","split":"test","body":"{\"choices\":\"['lose, because the advertisement was intended only as an invitation to make an offer.', \\\"lose, because the man did not notii' the store in writing that he intended to accept the offer.\\\", 'win, because the advertisement should be construed as a binding offer.', 'win, because it is immaterial whether the man was the first customer to appear at the store to purchase the necklace.']\",\"question\":\"A store published the following advertisement in a local newspaper on Monday, March 12:\\\"8 Brand New COWBOY HATS Beaver Felt,selling for $72. 50 . . . out they go . . . Sat. March 17,Each. . . $5. 1 Navajo Turquoise Necklace . . . worth $125, now selling for $40. \\\"FIRST COME, FIRST SERVED\\\"On the following Saturday, a man was the first person to arrive at Store and demanded the necklace. The store clerk refused to sell it to him, because it was a \\\"house rule\\\" that the necklace was intended for women only. If the man brings suit against the store for its refusal to sell him the necklace, the man will\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}