{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"26f15a4b-e2b7-5dae-9e56-7f68f3c974e7","task_key":"all--test--26f15a4b-e2b7-5dae-9e56-7f68f3c974e7","task_revision_id":"1","upstream_id":"","short_description":"A supermarket had just reopened after a two- week closing for renovations. On…","config":"all","split":"test","body":"{\"choices\":\"[\\\"the customer's answer was invited by the supermarket's attorney's question.\\\", \\\"the customer's answer was based on firsthand knowledge.\\\", \\\"the customer's answer was a statement of recent perception.\\\", \\\"the customer's answer was a present sense impression.\\\"]\",\"question\":\"A supermarket had just reopened after a two- week closing for renovations. On the morning of the reopening, a woman was shopping when she suddenly fell and suffered a broken leg. Thereafter, the woman sued the supermarket for personal injuries. In her suit, the woman claimed that she tripped over a paint bucket that had been lying in one of the aisles. At trial, the woman called another customer to testify that while he was shopping he heard a thud, turned around and saw the woman fall on the floor. He also testified that he saw an empty paint bucket lying a few feet from the woman's leg. On cross- examination, the customer got into an argumentative exchange with the supermarket's attorney. When asked his opinion of what caused the woman's fall, the customer testified, \\\"She fell because she tripped on the bucket. \\\" He admitted on cross-examination, however, that he didn't actually see the woman trip on the bucket. The supermarket's attorney then asked the customer the following question:\\\"If you didn't see the woman trip on the bucket, how do you know that she fell because of it?\\\" The customer answered, \\\"Because my girlfriend who was shopping with me told me later that she saw the woman trip over the bucket. \\\"The supermarket's attorney moves to strike the customer's last answer. If the trial judge overrules the defendant's motion and permits the customer's answer to stand, it will most likely be because\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}