{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"98b56438-1a5d-52c5-bdad-db9d858c0251","task_key":"default--validation--10175","task_revision_id":"1","upstream_id":"10175","short_description":"HellaSwag validation 10175","config":"default","split":"validation","body":"{\"activity_label\":\"Ice fishing\",\"ctx\":\"When they make a hole in the ice they put their fishing wire attached with bait into the hole and begin ice fishing until the woman catches a fish. after they\",\"endings\":\"['all get back in their boat they all begin to enjoy the cold water and in the ocean it gets really cold.', 'get returned they go back to fish digging under the ice and showing the fish they got there in the first place.', 'start they reel it in and we see the coming song on the screen.', \\\"'re done fishing, they are now at a wood fire and the fish is cooking on the wood.\\\"]\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}