{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"8028aeba-5e87-5c2f-a582-6d05d738415f","task_key":"default--validation--23584","task_revision_id":"1","upstream_id":"23584","short_description":"HellaSwag validation 23584","config":"default","split":"validation","body":"{\"activity_label\":\"Running a marathon\",\"ctx\":\"There's a woman talking about being the first woman to participate in a marathon in 1967. A picture of the participant when she was young is shown. she\",\"endings\":\"['talks about how different the marathons used to be back in the day where she shows clips of participants from a marathon at least 50 years ago.', \\\"shows her first woman's marathon she was two and a half years ago.\\\", 'then removes her warm sweatshirt, peels off her socks and legs and tosses them into the trash.', 'continues to talk as she rides around the track on her bike.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}