{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"864a6ef4-11e0-51d2-b347-0922fd7d0666","task_key":"default--validation--11181","task_revision_id":"1","upstream_id":"11181","short_description":"HellaSwag validation 11181","config":"default","split":"validation","body":"{\"activity_label\":\"Having an ice cream\",\"ctx\":\"Multiple images and clips of many different people eating and showing their ice cream. the bald man and the lady inside of the ice cream parlour bus\",\"endings\":\"['are shown working and picking up supplies as more images of patrons eat and show their ice cream treats.', 'speaks their hot eights while the bald man talks for a bit.', 'raise their mouths and raise their mouths to the camera with their own mouths.', 'then the front part of the ice cream shop and end by offering the food to one another and the closing credits flashing.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}