{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"4e38c194-c08d-5602-aef3-08f79ba724d6","task_key":"default--validation--11904","task_revision_id":"1","upstream_id":"11904","short_description":"HellaSwag validation 11904","config":"default","split":"validation","body":"{\"activity_label\":\"Plastering\",\"ctx\":\"A man in a yellow construction top is inside painting the wall beige. He goes from the bottom up and grabs more paint on his tool. he\",\"endings\":\"['is making progress across the wall pretty quickly the way he is doing it.', 'wraps the wall with yellow paper then cuts it with his ker tool.', 'applies the paint pads all over the wall and then removes the pieces before continuing.', 'goes from the top up to the top and back down again.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}