{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"7ca6edbf-69e7-5979-a0ed-9a06dcdc3766","task_key":"default--validation--49044","task_revision_id":"1","upstream_id":"49044","short_description":"HellaSwag validation 49044","config":"default","split":"validation","body":"{\"activity_label\":\"Playing harmonica\",\"ctx\":\"A man in a black pirate looking uniform in standing outside clapping and playing a kazoo. he\",\"endings\":\"['spins and flips in between the trees squeezing the handle and he turns his feet to the side to show his movements after spinning a few times.', 'take his long haired hair and blow dryer and start to blow dry it, he pulls back the hair with his finger and he is amazed by how much hair that happened.', 'claps his hands and dances around.', 'then appears and angles himself to the screen and begins playing drums with two sticks.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}