{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"979ae78a-3f1c-5773-9d15-260974571e73","task_key":"default--validation--41915","task_revision_id":"1","upstream_id":"41915","short_description":"HellaSwag validation 41915","config":"default","split":"validation","body":"{\"activity_label\":\"Gargling mouthwash\",\"ctx\":\"The man looks at the camera and then goes back to looking somewhere else and talking. then the cups they're both holding that has a blue liquid in it\",\"endings\":\"['go down their and shakes their hair out.', \\\"are shown and a second cup is brought down where they're all gathered together and the men begin to drink and enjoy it as one by one begin bungee jumping off of a very high height and tipping their heads back.\\\", 'holds at the top.', 'is then put into their mouths and they both gargle until they have lots of bubbles formed in their mouths.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}