{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"c764619d-721b-5c33-876a-dfd39723caa1","task_key":"default--validation--28725","task_revision_id":"1","upstream_id":"28725","short_description":"HellaSwag validation 28725","config":"default","split":"validation","body":"{\"activity_label\":\"Walking the dog\",\"ctx\":\"A cat is on the screen with a title of the video. A dog and cat are walking along a brick sidewalk. They pass some puddles and keep walking. the dog\",\"endings\":\"['stops to sniff the wall as the cat continues on.', 'and cat do occasional tricks towards each other as they pass the trash as they paw at it.', 'is pointing to the items in front of him.', 'then does off on its own on the sidewalk.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}