{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"07434403-f823-5cff-9c6a-5ddf12847505","task_key":"all--test--07434403-f823-5cff-9c6a-5ddf12847505","task_revision_id":"1","upstream_id":"","short_description":"There was an accident at a manufacturing plant located in California which state…","config":"all","split":"test","body":"{\"choices\":\"['Apply the punitive damage rule in California since that is where the accident occurred.', 'Apply the punitive damage rule in Washington since that is where the plaintiff resides.', 'Apply the punitive damage rule in Washington because that is where the court is located.', 'Let the jury in the case decide which state law of punitive damage should apply based upon their analysis of the relative culpability of the parties.']\",\"question\":\"There was an accident at a manufacturing plant located in California which state generally allows prevailing parties to collect punitive damages. A worker from Washington was injured at the California plant and she filed suit in federal district court in her home state of Washington. Washington does not generally allow punitive damages. The plaintiff has alleged in their federal lawsuit filed in Washington that they are entitled to punitive damages. The court should\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}