{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"871e2d1d-6036-5046-8d9f-7b5723ddb1e5","task_key":"default--validation--37662","task_revision_id":"1","upstream_id":"37662","short_description":"HellaSwag validation 37662","config":"default","split":"validation","body":"{\"activity_label\":\"Cleaning windows\",\"ctx\":\"Clips are shown of men hanging off of buildings washing windows while another man speaks to the camera. the men\",\"endings\":\"['wash windows all along a building and stop to smoke cigarettes and speak to one another.', 'blow the windows with spray and clean their hands thoroughly using first spray on each building, start again by spraying an electronic coating on every window as well as the sidewalk.', 'continue cleaning while a man blows leaves in front of them and is seen driving.', 'wash a car in a restaurant while the other speaks.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}