{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"9731f804-a6c0-50a6-ab32-47f8903f55aa","task_key":"default--validation--2647","task_revision_id":"1","upstream_id":"2647","short_description":"HellaSwag validation 2647","config":"default","split":"validation","body":"{\"activity_label\":\"Hand car wash\",\"ctx\":\"A colorful intro with a lot of flashing words flash across the screen along with a logo and a banner with foreign language written on it. a woman dressed in an orange and red dress and scarf\",\"endings\":\"['is sitting in front of a small sketched machine that is beginning to cook something and holds an egg over it that she adds onto the plate.', 'is standing and talking has she hold white paper in her hands.', 'is giving a video about shredded potato slicer ® treats and nutritious makeberry pudding.', 'is standing with a muffin and a small bowl of frnt and poke holes of the frnt, putting the frrt under the muffin before eating it.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}