{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"e335a241-f27e-5186-a62a-d8fa6f17c958","task_key":"default--validation--27352","task_revision_id":"1","upstream_id":"27352","short_description":"HellaSwag validation 27352","config":"default","split":"validation","body":"{\"activity_label\":\"Clipping cat claws\",\"ctx\":\"A close up of products are shown and leads into a woman holding and petting a cat. another women\",\"endings\":\"[\\\"then begins cutting the cat's claws and wrapping her up in a blanket.\\\", 'speaks to and pets the cat while the camera pans around cats and dogs and shows more dogs.', \\\"is then seen standing behind while the cat jumps on the woman's lap.\\\", 'sits with the cat while it pets her and a man speaks to the camera.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}