{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"c22ae127-4ffb-570e-a9d6-47df3827cfa7","task_key":"default--validation--40066","task_revision_id":"1","upstream_id":"40066","short_description":"HellaSwag validation 40066","config":"default","split":"validation","body":"{\"activity_label\":\"Family Life\",\"ctx\":\"[header] How to pick up an aggressive cat [title] Speak calmly and quietly. [step] Talk to the cat in a low, soft, soothing voice. Say its name, if you know it.\",\"endings\":\"[\\\"Pay attention to the cat's body language while talking to it. If the cat looks and acts normal, it may have calmed down.\\\", 'Use even, gentle tones, such as \\\" hi, cat. \\\" [title] Say \\\" give it a treat.', \\\"If the cat listens to you and is calming down you don't need to directly pick it up. If it is squirming or jumping around and won't let you pick it up without hissing or biting, it's trying to escape from you and you don't want to bother it.\\\", \\\"If it doesn't say anything in response, don't bother pressing or stroking it, as this could cause harm. [substeps] Be gentle, and don't pet the cat too much or it will get scared.\\\"]\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}