{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"8e361588-2c0f-5376-87cc-9088758adde7","task_key":"default--validation--5461","task_revision_id":"1","upstream_id":"5461","short_description":"HellaSwag validation 5461","config":"default","split":"validation","body":"{\"activity_label\":\"Playing harmonica\",\"ctx\":\"The player on the extreme right is playing a longer harmonica while the others are playing a smaller harmonica. they\",\"endings\":\"['hit the harmonica against the racket causing the player on the right to lose its tune.', 'are walking around as they play together, as if they are in sync and hoping the opponent will call a halt.', 'continue to play the big harmonica that is powered by 4 wires attached to it.', 'continue playing together as they dance along.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}