{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"381b64f9-4fb1-549f-ae58-4f5dd0c5dc00","task_key":"default--validation--46323","task_revision_id":"1","upstream_id":"46323","short_description":"HellaSwag validation 46323","config":"default","split":"validation","body":"{\"activity_label\":\"Ice fishing\",\"ctx\":\"A person is seen sitting on the snow around a hole and loading up a pole. He looks at a machine and continues adjusting his pole. he\",\"endings\":\"['uses a tool to pull up snow into the hole and continues to move around.', 'continues using the machine at all angles and showing off his show.', 'takes the stick out and attempts to speed up his push.', 'moves his pole around a bit and holds up a fish that he catches.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}