{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"a86e58e0-5c8b-50d8-8272-d974add22bb0","task_key":"default--validation--40569","task_revision_id":"1","upstream_id":"40569","short_description":"HellaSwag validation 40569","config":"default","split":"validation","body":"{\"activity_label\":\"Calf roping\",\"ctx\":\"Several men are shown at different rodeos riding horses and capturing the calf attached to the string. as more locations\",\"endings\":\"['appear it all fades to black.', 'are shown with riders riding horses and shots of other people getting briggs steers.', 'are shown, one final shot is shown of the older man.', 'are shown, more animals are captured and more people are shown watching the actions taking place.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}