{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"79595628-ba04-5278-a43a-2bc1a2f2ce64","task_key":"default--validation--27333","task_revision_id":"1","upstream_id":"27333","short_description":"HellaSwag validation 27333","config":"default","split":"validation","body":"{\"activity_label\":\"Ice fishing\",\"ctx\":\"A group of men labor over an ice fishing hole using a fishing pole, a wire, and their hands to finally wrangle a fish from deep under the ice. A group of men are standing over a small, round, hole in the ice. one man\",\"endings\":\"[\\\"picks up a rock and uses it to break the ice, attempting to break pieces of the fish' body as he struggles to get out of the hole.\\\", 'pulls and wrings out the slack line as others zale the fish from the hole.', 'pulls up water from the bottom and hands it to another man who pulls in more water.', 'is on his knees peering deeply into the hole while the other man has a fishing line submerged in the hole.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}