{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"fc79e136-ce9c-5e47-b6f9-e44aaa9a7e86","task_key":"default--validation--5404","task_revision_id":"1","upstream_id":"5404","short_description":"HellaSwag validation 5404","config":"default","split":"validation","body":"{\"activity_label\":\"Doing karate\",\"ctx\":\"A guy comes onto the screen to introduce a video about his character on a kids show. He begins to show several different clips from the show. he\",\"endings\":\"['explains another video that he and his friend play for fun while in a roller rink.', 'also comments about different aspects of the show while being interviewed.', 'continues to show how to properly paint the nails by using paintball guns.', 'then picks up some tagine and puts that on the screen, words flash across.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}