{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"57c000fc-732a-5ada-82e6-576b09a05ff6","task_key":"default--validation--11411","task_revision_id":"1","upstream_id":"11411","short_description":"HellaSwag validation 11411","config":"default","split":"validation","body":"{\"activity_label\":\"High jump\",\"ctx\":\"We see a man run and do a high jump 6'11 and see the replay. We then see the man jump at 6'9 and see the replay. the man\",\"endings\":\"[\\\"jumps at 6'11 again and we see the replay.\\\", \\\"runs back and upwards at 6'6'9 but then lands and returns to his feet.\\\", 'walks out of the helmet and we see his face again.', 'does his stroke over 6\\\\'4 \\\" and fades and we see the side screen again.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}