{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"17e308b9-914a-5592-a2c1-2a301b72bcb9","task_key":"all--test--17e308b9-914a-5592-a2c1-2a301b72bcb9","task_revision_id":"1","upstream_id":"","short_description":"An owner of a 3-story building, with apartments on the top two floors and a…","config":"all","split":"test","body":"{\"choices\":\"['arson only.', 'murder only.', 'arson and murder.', 'arson, burglary, and murder.']\",\"question\":\"An owner of a 3-story building, with apartments on the top two floors and a convenience store and gas station on the ground floor, resides with his family in one of the second-floor apartments. One day, a truck driver pulled into the gas station for some gas. After filling up the gas tank, the truck driver entered the convenience store for a bite to eat. He was eating a bowl of chili when he noticed a cockroach in the food. The truck driver demanded a new serving of chili. When the owner refused, the truck driver stormed out of the store in anger. The truck driver jumped into his truck and decided to get even by causing some damage. He purposely drove into one of the gas tanks, which ruptured the gas line. The collision caused a terrible explosion and the building became engulfed in flames. The owner could not escape the blaze and burned to death. At common law, the truck driver should be found guilty of\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}