{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"5013b0cc-7d03-577a-bb98-801e802ffd82","task_key":"default--validation--40477","task_revision_id":"1","upstream_id":"40477","short_description":"HellaSwag validation 40477","config":"default","split":"validation","body":"{\"activity_label\":\"Smoking hookah\",\"ctx\":\"She blows out all of the smoke that she inhaled, she is laying in bed by herself relaxing. She seems like she is very bored just blowing smoke recording herself. turns out\",\"endings\":\"['she is in lobotomized after that one eyeshadow song as she smokes all the smoke, then her mom gets up and comes out to smoke and help her smoke one more time.', \\\"to be what she is looking for, for the video still has you boys doing some stuff with her and you are both seated together in the living room recording her and illustrating the music she's listening to.\\\", 'the digital item records her driving very well.', ', there is another person in the room with her feeding her, she smiles really big and continues exhaling smoke out.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}