{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"120f40d0-9d39-59f0-a609-c3d04b376d25","task_key":"all--test--120f40d0-9d39-59f0-a609-c3d04b376d25","task_revision_id":"1","upstream_id":"","short_description":"A pedestrian domiciled in State A was crossing a street in State B when he was…","config":"all","split":"test","body":"{\"choices\":\"[\\\"File an answer raising the affirmative defense of contributory negligence and asserting a counterclaim for negligence, seeking damages for the driver's injuries.\\\", 'File an answer raising the affirmative defense of contributory negligence and defense of contributory negligence and move for judgment on the pleadings.', 'Move to dismiss for lack of personal jurisdiction, because the driver is not a citizen of State B.', 'Move to dismiss for lack of subject-matter jurisdiction, because the driver is not a U.S. citizen.']\",\"question\":\"A pedestrian domiciled in State A was crossing a street in State B when he was hit by a car driven by a citizen of a foreign country. Both the pedestrian and the driver suffered injuries. The pedestrian filed a negligence action against the driver in a federal district court in State B, seeking $100,000 in damages. The driver believes that the pedestrian was crossing the street illegally and is therefore responsible for the accident. The driver seeks an attorney's advice on how best to respond to the action. Assume that State B is a contributory negligence state. How should the attorney advise the driver to respond?\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}