{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"e38ee945-e9aa-56dc-90e7-226caea9c40c","task_key":"default--validation--40237","task_revision_id":"1","upstream_id":"40237","short_description":"HellaSwag validation 40237","config":"default","split":"validation","body":"{\"activity_label\":\"Getting a piercing\",\"ctx\":\"Various shots of a tattoo shop are shown followed by a woman sitting in a chair and a man getting items ready. She takes a sip of water and the man then pierces her lip with a needle. she\",\"endings\":\"['repeats the same procedure over and over again before nails are applied and all are shown.', 'smiles and moves her face and he stops and looks at the tattoo before tattooing her finger.', 'is shown again in a picture of her new piercing.', \\\"'s piercing her lip longer than the first time it happened and ends with her speaking.\\\"]\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}