{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"4f61aee0-621c-56fd-83e2-eadc59756622","task_key":"default--validation--23360","task_revision_id":"1","upstream_id":"23360","short_description":"HellaSwag validation 23360","config":"default","split":"validation","body":"{\"activity_label\":\"Playing harmonica\",\"ctx\":\"Three men are sitting on a couch with the man in the middle smiling, the man on the right playing a harmonica and the man on the far left holding his finger to his upper lip area and making a noise with his bottom lip by plucking it with his hand. The man on the right taps his foot as the two men play music. the\",\"endings\":\"['man in the middle is really good at it but the man playing now is blocked as he is spinning is motion left and right.', 'camera pans to an onlooker standing on the side of the room watching and smiling.', 'man on the right removes his palm and uses the purple slide screen.', 'men continue to play as the man, the crooner on the left, with the voice down from the music might be able to hear him.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}