{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"0cbf3bf2-39a2-5ae4-91e2-4169b3b102cf","task_key":"default--validation--5232","task_revision_id":"1","upstream_id":"5232","short_description":"HellaSwag validation 5232","config":"default","split":"validation","body":"{\"activity_label\":\"Ice fishing\",\"ctx\":\"The boy struggles to get the fish out and someone comes and takes it out for him. Now there are four men visible and they all begin to fish out of the ice pond. eventually\",\"endings\":\"['one comes out and gets a fish and places it on the ice before getting a fish to crawl out afterwards.', 'they are all in the pond on fish cage hooks doing tricks.', 'one holds a fish over the hole and once he gets it down he swims towards the camera.', ', a man grabs a chair and starts fishing and taking his fish off of his line.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}