{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"36be2263-3d6d-5a3c-b476-5debff5e9daf","task_key":"default--validation--38516","task_revision_id":"1","upstream_id":"38516","short_description":"HellaSwag validation 38516","config":"default","split":"validation","body":"{\"activity_label\":\"Gargling mouthwash\",\"ctx\":\"A woman is seen speaking to the camera and leads to her holding up mouth wash and then swishing it around her mouth. she\",\"endings\":\"['continues swishing it around while closing the cap then spits it out and smiles to the camera.', 'then puts the mouth wash in her mouth and then turns the tub on causing the water to hot.', 'looks back to the camera and takes a drink out of the mouth wash.', 'then grabs a large snobbish brush and begins soaping her face and wiping down her face.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}