{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"4e46187c-52d9-5c55-a59e-cc42b1d35117","task_key":"default--validation--27402","task_revision_id":"1","upstream_id":"27402","short_description":"HellaSwag validation 27402","config":"default","split":"validation","body":"{\"activity_label\":\"Computers and Electronics\",\"ctx\":\"[header] How to hold a cat by the scruff [title] Remove any scents from your body that may be offensive to your cat. [step] Strong-smelling perfumes or colognes may be bothersome to him. The scent of a dog may be especially unsettling.\",\"endings\":\"[\\\"The scent of a cat who goes inside his home might soothe him, take his mind off possible smells and keep him away from potentially unpleasant smells. [substeps] If this's the case, smell only pheromones from your cat.\\\", \\\"It's always best to get rid of strong smelling fragrances, such as garlic, ginger, and cinnamon, before he enters your home. [substeps] The urinary tract secretes the fatty deposits that give a cat the scent he needs.\\\", '[title] Allow your cat to become comfortable with you before you try to scruff him. [step] Calmly petting him and allowing him to rub up against your hand will relax him.', 'If your cat has been marking you, it may be essential to remove the pheromone wash from his fur. [substeps] If you have tried and nothing results, then go back into the house and try again.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}