{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"88959ad5-2a39-5dc5-af2f-916c2217b478","task_key":"default--validation--37670","task_revision_id":"1","upstream_id":"37670","short_description":"HellaSwag validation 37670","config":"default","split":"validation","body":"{\"activity_label\":\"Computers and Electronics\",\"ctx\":\"[header] How to sew a sock dog [title] Select a suitable sock. [step] You'll need a good quality, long sock. It needs to be long, to accommodate the dog's body.\",\"endings\":\"['The longer, the better as it will provide all the pieces you need. [substeps] If the sock proves to be too short, you will need to use scrap materials for the ears, tail and legs.', 'For the sock dog, choose one big, one medium size sock, and 1 mini sock, for a pair of socks made to fit all sizes. As you become more frustrated, start looking around.', '[substeps] While a true sock dog might be able to dance and do most odd cartwheels, there are some knots that can make your dog hop up in the air as if it was the witch running around town instead. The o and o solution mixture is easily available on the internet.', \\\"[title] Prepare the sock pup's lining. [step] Lay the sock pup down on a clean surface and cut off the very edge of the paper, so that you have a piece that matches the sock.\\\"]\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}