{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"dd3fe8f3-5151-5b24-b0c0-e0d278cb15b4","task_key":"default--validation--21578","task_revision_id":"1","upstream_id":"21578","short_description":"HellaSwag validation 21578","config":"default","split":"validation","body":"{\"activity_label\":\"Disc dog\",\"ctx\":\"A teen boy is running on the beach. A dog is chasing him, trying to catch his frisbee. he\",\"endings\":\"['gets one and throws it and he and his dog chase after it.', 'throws the frisbee in the dog, but the dog hits it and runs off.', 'throws the frisbee and the dog catches it one time.', 'throws the frisbee, and the dog jumps into the air to catch it.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}