{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"4a5cbc7b-7b27-5172-8670-aabf2dfd483f","task_key":"default--validation--29843","task_revision_id":"1","upstream_id":"29843","short_description":"HellaSwag validation 29843","config":"default","split":"validation","body":"{\"activity_label\":\"Cars & Other Vehicles\",\"ctx\":\"[header] How to open frozen car doors [title] Push on your car door. [step] Apply pressure by leaning on your frozen door. Push as hard as you can against the door.\",\"endings\":\"['Turn your foot inward to gain pressure on the door. [substeps] If your door is locked, push backward on the door for five seconds, rotating your foot outward to get inside the door.', 'Stay until you hear your car doors click into place and press harder. [title] Close your car doors if your car door has button locks or screws.', \\\"The pressure could break the ice around the door's seal, enabling you to open the door. [substeps] This section assumes you can unlock your car, but not open it.\\\", 'If you can still get in the car, put your stick through the door, then push the door all the way open. [substeps] Try sliding the door all the way open by jamming the stick in the door, tightening the grips, and then pushing the door all the way open.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}