{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"b0ea71f9-cca8-52a5-b9c4-f5531942a8ad","task_key":"all--test--b0ea71f9-cca8-52a5-b9c4-f5531942a8ad","task_revision_id":"1","upstream_id":"","short_description":"The retributive theory of punishment does not claim that","config":"all","split":"test","body":"{\"choices\":\"['wrongdoers deserve to be punished.', 'specific punishments are justified based on the overall intrinsic value that they produce', 'the punishment should fit the crime.', 'the death penalty might be morally justified in some cases of murder.']\",\"question\":\" The retributive theory of punishment does not claim that\",\"subject\":\"moral_disputes\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}