{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"63ef186d-7fc1-51e0-a8e1-5248f212ae22","task_key":"default--validation--15025","task_revision_id":"1","upstream_id":"15025","short_description":"HellaSwag validation 15025","config":"default","split":"validation","body":"{\"activity_label\":\"Fixing the roof\",\"ctx\":\"A playground is shown and a back yard as the camera pans over to a window on a roof. the video\",\"endings\":\"['shows various pictures on how to fix a roof and the man pans over the window again with material.', 'then goes back to the \\\" backyard \\\" and starts with this lone young boy using a small shovel to lift up the old roof.', 'ends with the man showing other children how to install some sort of roof using metal paper, hammers and metal bars.', 'returns to the man shoot while again the man is sitting.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}