{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"751f4f1b-d493-5e04-9ca2-04447971d46c","task_key":"default--validation--10667","task_revision_id":"1","upstream_id":"10667","short_description":"HellaSwag validation 10667","config":"default","split":"validation","body":"{\"activity_label\":\"Ice fishing\",\"ctx\":\"One of the men comes out of his camping tent with a fish in his hand. The two men are sitting in their make shift tent, fishing through a hole in the frozen lake. they\",\"endings\":\"['are patiently sitting with their fishing rods till they fish out mid size fish from the holes.', 'come out to fish from the ice, looking back as they go down the mountain.', 'then try and catch a seal, but it is too fish ridden for them to finally get one in their mouths.', 'fish up a large red hole while wearing protective tape over their mouths.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}