{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"027f8c6c-7c68-5f19-ba06-4bfd60b45b9b","task_key":"default--validation--42034","task_revision_id":"1","upstream_id":"42034","short_description":"HellaSwag validation 42034","config":"default","split":"validation","body":"{\"activity_label\":\"Computers and Electronics\",\"ctx\":\"[header] How to open a bostitch heavy duty stapler [title] Set your bostitch stapler on a hard, flat surface. [title] Look for the back assembly, sometimes called a pusher. [step] On some models this will have a red tab pointing out of the back.\",\"endings\":\"['On others, it may have a stack that points toward the side, depending on the model. [substeps] Open the front assembly and pull back on the back assembly--if the tray is wooden, it will lock.', \\\"In newer models it may be colored so it's hard to see it. [title] Thread your arm through the pusher.\\\", '[title] Hold the top of the stapler in place with your non-dominant hand. [step] Never place your hand under the stapler.', 'In other models it will have a silver tab. [title] While the front assembly looks kind of like a regular one, it\\\\'s usually labeled \\\" laformation \\\" with a hole from which a needle can sit.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}