{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"90db3dc9-4b9e-52d9-bd68-0d0607f807ee","task_key":"default--validation--22513","task_revision_id":"1","upstream_id":"22513","short_description":"HellaSwag validation 22513","config":"default","split":"validation","body":"{\"activity_label\":\"Pets and Animals\",\"ctx\":\"[header] How to train an untamed dog [title] First, name the dog. [step] The name should be something short and easy to say, but don't pick a word you use a lot in daily conversation. [title] When you've named the animal, say the name out loud whenever you interact with the dog.\",\"endings\":\"[\\\"[step] Make sure it doesn't get confused with another word, though. [title] Once you've named the animal, make sure you use a frequently-used word in daily conversation.\\\", \\\"[step] Soon after it is introduced to you with the name, say it again if it doesn't know the word yet. After it knows, say the full word again if it follows your command.\\\", \\\"[step] If the dog isn't good at paying attention, try saying the dog's name and then immediately giving it a treat. Very soon the dog will learn to focus on you when it hears its name.\\\", \\\"[step] You should be able to pronounce the name correctly all the time! [title] Training treats allows you to get a dog and, if the training is successful, encourage it to stay. [step] If the dog doesn't stay, take it to your local humane alley or the petting store.\\\"]\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}