{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"ed59ebfe-1964-57d9-a5d4-ac2170cf00f2","task_key":"default--validation--36810","task_revision_id":"1","upstream_id":"36810","short_description":"HellaSwag validation 36810","config":"default","split":"validation","body":"{\"activity_label\":\"Pets and Animals\",\"ctx\":\"[header] How to train a rottweiler to be a guard dog [title] Start with the \\\" sit \\\" command. [step] Take your rott to a quiet, calm room and get its attention by showing it a treat or a toy. Using a calm, friendly tone, say \\\" sit \\\" as you raise the treat or toy above your rott's head.\",\"endings\":\"['This will be your cue for the behavior. Your rott should follow your commands without looking down at the treat or toy.', \\\"As your rott follows what you have in your hand, it will lose its balance and sit its butt on the floor. Once they're sitting, immediately give them the treat or toy.\\\", 'You want to lay your desired distance between you and your rott, so move away from it, position your hand, and then lower your hand quickly. This gradually helps your rott learn this command.', 'The desired behavior will trigger your rott to sit. [title] Use a clicker to reinforce your commands.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}