{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"4cbe2fec-9331-5f92-a419-953b61d69438","task_key":"default--validation--18040","task_revision_id":"1","upstream_id":"18040","short_description":"HellaSwag validation 18040","config":"default","split":"validation","body":"{\"activity_label\":\"Family Life\",\"ctx\":\"[header] How to celebrate your husband's birthday [title] Collect ideas throughout the year. [step] Pay attention when he casually mentions something he likes or needs. Maybe he gets excited about a new book coming out by his favorite author.\",\"endings\":\"[\\\"If something is bothering him, your life could be impacted by a rose print under his bed. Don't overwhelm your husband with everything that strikes his fancy and go crazy trying to gather inspiration.\\\", 'Maybe he just has a major schedule or a few errands that he barely has time to finish. Think about what he might need when you are planning a surprise party.', \\\"Maybe he's come up with an idea he wanted just before a sleepover, or maybe he wanted something before seeing one of his friends. These will all serve to give you a head start on things.\\\", \\\"Maybe he gets frustrated with the broken zipper on his old laptop case. Make note of these little clues and write them down so you'll have some inspiration when his birthday rolls around.\\\"]\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}