{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"1bc2ee42-5337-551a-85ed-dc7e202559ca","task_key":"default--validation--15107","task_revision_id":"1","upstream_id":"15107","short_description":"HellaSwag validation 15107","config":"default","split":"validation","body":"{\"activity_label\":\"Calf roping\",\"ctx\":\"The man lassos the calf, jumps off the horse, throws the calf on its side and quickly ties it up. the man\",\"endings\":\"['gets up, walks back to his horse, jumps back on and the horse slowly begins to walk as the calf is trying to stand up.', 'keeps tying up the calf, sincgars it and stands up.', 'walks off the horse and grabs a small rock.', 'gets on his horse and camp on top of a cliff.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}