{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"2a757e23-5f76-5cab-b7f8-a6d19181af0f","task_key":"default--validation--6510","task_revision_id":"1","upstream_id":"6510","short_description":"HellaSwag validation 6510","config":"default","split":"validation","body":"{\"activity_label\":\"Removing ice from car\",\"ctx\":\"The man then comes outside and starts talking about the snow holding a red plastic stick in his hand. with the stick\",\"endings\":\"['he begins to add and clip to the tire on his car.', ', he proceeds to the car and start slicing the snow off of his windshield.', 'that is attached to an attached bucket he begins explaining how to sled and why he is being so paid for for his hard work.', 'he has some snow on his shovel to serve against the snow.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}