{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"9ac4682c-fcca-5a50-9997-2153e9eb8470","task_key":"default--validation--25362","task_revision_id":"1","upstream_id":"25362","short_description":"HellaSwag validation 25362","config":"default","split":"validation","body":"{\"activity_label\":\"Horseback riding\",\"ctx\":\"To the left of them, there is a barbwire fence with two goats on the side, a playground, and several people sitting under the hut. as they\",\"endings\":\"['approach, one of the goats charges the other and the man is rolled off the back.', 'continue their expedition, more people join them and more buildings of the village become vivid.', 'turn they see the goats looking at them as the people on the left grinder their goats.', 'approach the hut there is a family of dogs and they see a man on the ground also running on the sidewalk and swings by himself.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}