{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"1fd73f04-f423-5868-8ea8-e3e2e4192a74","task_key":"all--test--1fd73f04-f423-5868-8ea8-e3e2e4192a74","task_revision_id":"1","upstream_id":"","short_description":"A rider entered a subway car at the 42nd Street station. Because all of the…","config":"all","split":"test","body":"{\"choices\":\"['liable for battery.', 'liable, because the rider believed that the girl consented to the contact under the circumstances.', \\\"not liable, because the rider's conduct was socially acceptable under the circumstances.\\\", 'not liable, because the girl was not harmed by the contact.']\",\"question\":\"A rider entered a subway car at the 42nd Street station. Because all of the seats were occupied, the rider stood in the subway car and grabbed a pole to secure his balance. As the subway car was proceeding downtown, the rider glanced at a girl standing next to him. Suddenly, the subway car made an unexpected stop. The rider momentarily lost his balance and grabbed the girl around the waist (to avoid falling). Once the rider regained his balance, he removed his hands from the girl's waist and grasped the pole again. In a civil action instituted by the girl against the rider, he will most likely be found\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}