{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"4888a027-92ea-5222-a196-26c1f16134ec","task_key":"default--validation--47142","task_revision_id":"1","upstream_id":"47142","short_description":"HellaSwag validation 47142","config":"default","split":"validation","body":"{\"activity_label\":\"Slacklining\",\"ctx\":\"A long rope is shown with a man tightening it and a woman walking across holding onto a man. more people\",\"endings\":\"['are shown riding on the rope as well as laying on the side and swimming in the water.', 'are shown stretching on the sides and then the man pushes the rope uneven.', 'are seen unraveling the rope while people stand and watch them.', 'attempt to walk the rope while others watch on the side.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}