{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"44813494-1392-52ba-b3e0-ef746322389a","task_key":"default--validation--30013","task_revision_id":"1","upstream_id":"30013","short_description":"HellaSwag validation 30013","config":"default","split":"validation","body":"{\"activity_label\":\"Gargling mouthwash\",\"ctx\":\"The man brings the bottle closer to his mouth and laughs some more and then pours the liquid from the small bottle onto his mouth and begins to gargle until bubbles come out of his mouth. the man\",\"endings\":\"['grabs more water and continues trying to gargle and spit some more.', 'smiles while backing away and wiping his mouth with his hand.', 'moves his body a little bit and spits on the empty bottle.', 'then quickly leans forward and spits it all out of his mouth.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}