{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"82feed0f-dafd-54af-aca9-3027c26e380c","task_key":"default--validation--15060","task_revision_id":"1","upstream_id":"15060","short_description":"HellaSwag validation 15060","config":"default","split":"validation","body":"{\"activity_label\":\"Ice fishing\",\"ctx\":\"A large fish is seen swimming under frozen water. A man creates a hole in the ice to catch the fish. he\",\"endings\":\"['catches it on a line, and pulls until he is able to get it through the hole.', 'throws it down the hole with dogs and the last image is taken at night.', 'puts the catch in his mouth and is shown smiling to the camera.', 'jams and seals the hole with a long piece of fishing paper.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}