{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"ca88b492-b8e6-5858-9160-cabf96b7426e","task_key":"all--test--ca88b492-b8e6-5858-9160-cabf96b7426e","task_revision_id":"1","upstream_id":"","short_description":"Carol Collector was a serious antique car buyer, and was always searching for…","config":"all","split":"test","body":"{\"choices\":\"['Carol, because the internet picture was of a car that had never been driven, not the actual used car she was to buy.', 'The seller, because the buyer was aware the Thunderbird was sold \\\"as is.\\\"', 'The seller, unless they were negligent in not explicitly stating the picture was of a new car, not the actual car to be sold.', 'Carol, because the \\\"as is\\\" disclosure was not totally effective.']\",\"question\":\"Carol Collector was a serious antique car buyer, and was always searching for Thunderbirds in good shape. She saw a newspaper ad offering a 1964 Thunderbird sports coupe for $25,000, The ad also gave a website address \\\"to view a picture of the car,\\\" which Carol visited, and which showed a T-Bird with a perfect body and interior. Carol paid the $25,000 and signed a sales contract which specified that the Thunderbird was \\\"used and sold as is.\\\" When Carol went to pick up the car, she learned it was a 1968 model, which was not at all rare and worth much less than the advertised 1964 model. If Carol brings suit against the seller, the likely outcome is for\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}