{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"ac4b5174-7219-5788-aef4-eba21bd6416d","task_key":"default--validation--33742","task_revision_id":"1","upstream_id":"33742","short_description":"HellaSwag validation 33742","config":"default","split":"validation","body":"{\"activity_label\":\"Home and Garden\",\"ctx\":\"[header] How to protect doors from dog scratches [title] Use a scratch screen to absorb the damage. [step] Similar to a screen protector for your phone, scratch screens go on top of your door and absorb any damage caused by your dog. You can purchase plastic scratch screens at certain furniture and pet stores, or you can make a more durable screen at home by cutting out a sheet of fiberglass and fixing it to your door.\",\"endings\":\"['[substeps] Look for plastic scratch screens made from rubber or metal. These may work better for both your door and your home.', '[title] Project a note to yourself requesting a scratch screen. [step] This will help you remember to call/text your landlord.', \\\"[substeps] Over time, scratch marks will begin to appear on the scratch screen itself. This indicates that it's time to replace the screen.\\\", '[substeps] Hold your scratch screen at a 45 degree angle to your door and brace it against the door for 5 to 10 seconds. Then, hold it at a 45 degree angle to the door.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}