{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"5dbc00b0-5c56-5745-b627-b18c04ae3287","task_key":"default--validation--23359","task_revision_id":"1","upstream_id":"23359","short_description":"HellaSwag validation 23359","config":"default","split":"validation","body":"{\"activity_label\":\"Playing harmonica\",\"ctx\":\"Three men are sitting on a couch with the man in the middle smiling, the man on the right playing a harmonica and the man on the far left holding his finger to his upper lip area and making a noise with his bottom lip by plucking it with his hand. the man on the right\",\"endings\":\"['also makes a funny noise with his mouth as he continues holding his finger in his mouth.', 'taps his foot as the two men play music.', 'is playing the harmonica and doing another tune with his fingers moving back and forth between the keys.', 'holds up a bottle of red wine and pours it down the person\\\\'s top and the man in the middle takes a sip, spits it out and laughs \\\" if you have had your family and friends blown away and you haven\\\\'t had them visited they\\\\'re probably jumping up and cheering for you.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}