{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"da16d371-a153-57b3-94a5-8bebf1cba9de","task_key":"default--validation--4661","task_revision_id":"1","upstream_id":"4661","short_description":"HellaSwag validation 4661","config":"default","split":"validation","body":"{\"activity_label\":\"Bathing dog\",\"ctx\":\"There is another little puppy running around the dog and a little kitten too. The woman takes some liquid soap and scrubs it on the dog. after she\",\"endings\":\"['is done with the dog she puts him down and questions him again.', \\\"'s done, she throws away the soap and the kitten is happy.\\\", \\\"'s done washing, she wipes him dry with a towel.\\\", 'has cleaned the bottom of the dog, the woman grabs a piece of paper and puts the shirt back on.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}