{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"5e947565-1c73-58ba-af0f-03c71cf572da","task_key":"default--validation--40715","task_revision_id":"1","upstream_id":"40715","short_description":"HellaSwag validation 40715","config":"default","split":"validation","body":"{\"activity_label\":\"Ice fishing\",\"ctx\":\"A man dressed in winter gear is sitting out in winter on a frozen lake. He is sitting there with fishing equipment fishing on the frozen lake. he\",\"endings\":\"['gets up and continues to do stunt fishing as the cold continues to sink into his clothes.', 'then pulls the fish deeper into some ice the the continue to fish.', 'uses a bait to fish and catches a mid size gray fish.', 'holds a bag that is attached to his horse by a rope as he fishes and he is still mooning from the frozen lake.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}