{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"e25a522f-3f93-5ba0-ac7f-931f6bb4fc0f","task_key":"default--validation--17830","task_revision_id":"1","upstream_id":"17830","short_description":"HellaSwag validation 17830","config":"default","split":"validation","body":"{\"activity_label\":\"Playing harmonica\",\"ctx\":\"Four women are in the studio playing harmonica. As the women play they blow the harmonica and clap their hands on the instruments. the women\",\"endings\":\"['continue to play for the audience outside.', 'are moving their bodies slightly as the play the harmonica.', 'continue to blow the harmonica and clap their hands one another.', 'split apart and continue serving the harmonica while the others continue screwing in a certain direction and playing faster.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}