{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"54ab7562-ea44-55ed-bbb9-1144c64301ff","task_key":"default--validation--26369","task_revision_id":"1","upstream_id":"26369","short_description":"HellaSwag validation 26369","config":"default","split":"validation","body":"{\"activity_label\":\"Walking the dog\",\"ctx\":\"A large cat is walking down a brick pathway with a leash alongside a small white dog. as the walk\",\"endings\":\"['continues, the cat stops and begins staring at a parked car with large red flames painted on the side.', ', the dogs nose brushes the leash.', ', the dog pushes the woman on its front legs.', ', the black and white perform various turns and tricks with the small white dog.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}