{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"b8b772fc-a6ab-5bea-ba8c-6b424dde97bc","task_key":"all--test--b8b772fc-a6ab-5bea-ba8c-6b424dde97bc","task_revision_id":"1","upstream_id":"","short_description":"Doug Dogman had a domesticated golden retriever named Rover. Rover was a good,…","config":"all","split":"test","body":"{\"choices\":\"[\\\"Ginny, because Doug's dog was a cause in fact of Ginny's injuries.\\\", 'Doug, because Retriever is a domesticated animal.', 'Doug, unless a statute or ordinance bans unleashed dogs on public streets.', 'Doug, because the strict liability rule for dogs would not include dogs not displaying vicious propensities.']\",\"question\":\"Doug Dogman had a domesticated golden retriever named Rover. Rover was a good, friendly dog who loved to run and chase cars but had never hurt anyone. One dark evening before the street lights turned on, Rover was chasing cars in the street, and Ginny Grandmother, aged 85, swerved to avoid hitting the dog. As she swerved, Ginny hit a parked car and suffered a head injury. If Ginny brings suit against Doug Dogman under a strict liability theory, the court will likely find for\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}