{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"445728c3-bcd8-5d2a-9189-967237c185c0","task_key":"default--validation--27993","task_revision_id":"1","upstream_id":"27993","short_description":"HellaSwag validation 27993","config":"default","split":"validation","body":"{\"activity_label\":\"Having an ice cream\",\"ctx\":\"A close up of a box of ice cream is shown and leads into a bowl of ice cream sitting beside the box. the camera\",\"endings\":\"['continuously pans around to all the scoops she and ice cream are using as well as the different scoops of ice cream the girl is using.', 'leads into the bowl and turns towards a bearded man eating the ice cream and nodding in approvement.', 'is captured close up as well as a woman grabbing the ice cream in her mouth.', 'zooms in over the bowl and shows a mountain covered with pureed scoops.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}