{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"58114752-30d8-51a0-9410-8114803fb432","task_key":"all--test--58114752-30d8-51a0-9410-8114803fb432","task_revision_id":"1","upstream_id":"","short_description":"A construction worker was working at the construction site of a new building. An…","config":"all","split":"test","body":"{\"choices\":\"['No, because such risks are inherent in construction work.', 'No, because the worker was not in danger while on the stalled elevator. while on the stalled elevator.', 'Yes, because the elevator stalled due to a manufacturing defect.', 'Yes, because the worker was falsely imprisoned in the stalled elevator.']\",\"question\":\"A construction worker was working at the construction site of a new building. An open elevator, which had been installed in the building by the elevator manufacturer, was used to haul workers and building materials between floors. While the worker was riding the elevator, it stalled between floors due to a manufacturing defect in the elevator. The worker called for assistance and was in no danger, but after waiting 15 minutes for help, he became anxious and jumped 12 feet to get out. He severely injured his back when he landed. In an action by the worker against the elevator manufacturer to recover for his back injury, is the worker likely to obtain a judgment for 100% of his damages?\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}