{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"8fe6bb8c-b164-5b49-b45d-dc2488cdb1d4","task_key":"default--validation--3659","task_revision_id":"1","upstream_id":"3659","short_description":"HellaSwag validation 3659","config":"default","split":"validation","body":"{\"activity_label\":\"Playing violin\",\"ctx\":\"A violinist, lindsay stirling is performing live with some other participants on a stage in front of an audience. she\",\"endings\":\"['begins by playing the violin while the other performers dance in a choreographed fashion.', 'is giving brief commentary on her performance.', 'sings her song right into the microphone followed by dancing gracefully and holding her body up and facing the audience as she finishes.', 'is blowing large amounts of smoke while on a set of drums and a guitar as some people watch her on a stage.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}