{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"7121dc6c-20a1-549f-8040-55d78a1992a7","task_key":"default--validation--5226","task_revision_id":"1","upstream_id":"5226","short_description":"HellaSwag validation 5226","config":"default","split":"validation","body":"{\"activity_label\":\"Pets and Animals\",\"ctx\":\"[header] How to encourage multiple cats to get along with each other [title] Allow the cats to smell each other before meeting. [step] Keep the new cat in a separate room so the other cats can smell her through the door. You can use your bathroom if you don't have a spare room.\",\"endings\":\"[\\\"Using a pheromone diffuser or blow dryer will help your cats get acquainted with each other's scent. [title] Keep all food, water, and toys in a separate room.\\\", \\\"[substeps] When you let your cat in the room, let her sniff the other cat and climb up to the cat's shoulder. As she grows into a more dominant cat, there is a slight gap between their private rooms.\\\", 'When they do meet but don\\\\'t get in the room, put the door to your room off to the side, so the cats can scent each other. [substeps] Say, \\\" this is a nice kitty, but make sure they won\\\\'t fight over where to hide.', \\\"[substeps] Give the established cat something with the new cat's scent on it to sleep on. The ideal object is a t shirt with your scent on it that the new cat has sat on.\\\"]\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}