{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"2d43bcce-6d72-5245-8c0d-c6c41c5f5760","task_key":"default--validation--33118","task_revision_id":"1","upstream_id":"33118","short_description":"HellaSwag validation 33118","config":"default","split":"validation","body":"{\"activity_label\":\"Playing harmonica\",\"ctx\":\"The lady continues playing the harmonica as the elevator moves further and stops at other levels and more people walk in. people\",\"endings\":\"['wrench the ramps and levels as the camera follows their movements that lead to the lady hitting harmonies and some hokey syllables.', 'are moving in a line as the lady plays.', 'start to take pictures of the harmonica as they walk through the lobby and outside to the bike repair shop.', 'continue to get on and off the elevator but she continues to play her harmonica.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}