{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"5dfa80da-5222-5950-b54b-83c26d2d731c","task_key":"default--validation--18958","task_revision_id":"1","upstream_id":"18958","short_description":"HellaSwag validation 18958","config":"default","split":"validation","body":"{\"activity_label\":\"Painting furniture\",\"ctx\":\"A man is standing with his legs apart and is holding a sprayer in his right hand and spraying a dresser with many drawers as a man walks past him and looks at what he's doing. The man brings his legs closer together while still spraying and he begins closing the drawers. the man\",\"endings\":\"['stops and looks at the camera and is now getting excited.', 'then begins bending all over and showing off his back collar.', 'turns his body towards the drawer and is shaving in front of him while shaving with the sprayer.', 'then goes to the right side of the dresser and sprays there, then goes back to the front, then the top, and then on the left side.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}