{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"dd07fe9b-cfbf-5b61-8ec4-f72e271b92bd","task_key":"all--test--dd07fe9b-cfbf-5b61-8ec4-f72e271b92bd","task_revision_id":"1","upstream_id":"","short_description":"A customer in a restaurant ordered a bowl of the restaurant's famous homemade…","config":"all","split":"test","body":"{\"choices\":\"['recover, if a reasonable consumer would not expect the presence of such a pebble in the chowder.', 'recover, but only if the restaurant had received prior notice of the defective condition of the soup.', 'not recover, because a reasonable inspection of the soup would not have disclosed the existence of the pebble.', 'not recover, because the soup would not constitute an unreasonably dangerous product when it was served to the customer.']\",\"question\":\"A customer in a restaurant ordered a bowl of the restaurant's famous homemade Manhattan clam chowder. While eating the chowder, the customer broke a tooth on a pebble in the soup, which a reasonable inspection would not have discovered. When the customer complained to the restaurant's owner, the owner admitted that the chowder was not, in fact, homemade, but that it had been poured from a can of chowder purchased from a wholesale distributor of food products. In a strict liability action by the customer against the restaurant, he will most likely\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}