{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"674c92d9-412b-50e0-8e06-97775c597e5f","task_key":"all--test--674c92d9-412b-50e0-8e06-97775c597e5f","task_revision_id":"1","upstream_id":"","short_description":"On his 21st birthday, a son enlisted in the army. As he was about to leave for…","config":"all","split":"test","body":"{\"choices\":\"[\\\"succeed, because the son's promise to refrain from drinking alcohol provided sufficient consideration.\\\", \\\"succeed, because the son's promise to refrain from taking illegal drugs and drinking alcohol provided sufficient consideration.\\\", \\\"not succeed, because the father's promise was only a conditional gift.\\\", \\\"not succeed, because the father's promise was void.\\\"]\",\"question\":\"On his 21st birthday, a son enlisted in the army. As he was about to leave for eight weeks of basic training, his father said, \\\"I will buy you a new car at the end of basic training if you promise me not to take illegal drugs or drink alcohol while at basic training. \\\" The son replied, \\\"You can order the new car right away. I promise not to take illegal drugs or drink alcohol, as you requested. \\\"At the end of basic training, the son was given a leave and returned home. His father asked, \\\"Did you abide by your promise not to take illegal drugs or drink alcohol?\\\" The son replied, \\\"Yes, Father. \\\" The father then told the son that he had already ordered the new car and that it would be available for delivery within one month. One week later, the son received the tragic news that his father had died suddenly. At the funeral, the executor of the father's estate told the son that he did not feel compelled to give the son the newly arrived car. In a suit against the executor of the father's estate to recover the new car, the son will\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}