{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"b80fd963-e947-5d6f-9acf-aa5250539cc7","task_key":"default--validation--38287","task_revision_id":"1","upstream_id":"38287","short_description":"HellaSwag validation 38287","config":"default","split":"validation","body":"{\"activity_label\":\"Running a marathon\",\"ctx\":\"Three women are seen speaking to the camera and answering questions that the hosts ask. Clips are shown of people running down the street while the women continue to speak. the girls\",\"endings\":\"['stand and talk to the camera with one holding a phone and the other putting a contact lens in her eye.', 'then begin performing martial arts moves on one another and ends with the girls clapping exuberantly.', 'talk more and more while people are still shown running down the road.', 'walk away and resume speaking while moving their arms.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}