{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"7a48b481-ba7e-5335-8525-5439605a7381","task_key":"default--validation--28351","task_revision_id":"1","upstream_id":"28351","short_description":"HellaSwag validation 28351","config":"default","split":"validation","body":"{\"activity_label\":\"Food and Entertaining\",\"ctx\":\"[header] How to train your new dog to respond to a new name [title] Associate the new name with the old name. [step] For the first week, you may need to call the dog both names. To do this, say the old name and the new name together.\",\"endings\":\"['For example, you can say \\\" trackty \\\" \\\" great dog, \\\" \\\" mad dog, \\\" \\\" fine dog, \\\" \\\" big dog, \\\" and \\\" bad dog \\\" combined. You can also modify the introductions while your new dog is still a puppy.', '[substeps] Have the kids help with this process. Have them call each name on an ipad.', 'Continue for a week or two, then switch. [title] Ask your dog to get used to the new name.', 'If the dog responds to this compound-name, you can start dropping the old name after about a week. [substeps] For example, if the dog\\\\'s old name is \\\" wolfgang \\\" and their new name is \\\" charlie, \\\" you should call them \\\" wolfgang charlie.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}