{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"00f6012a-6aee-566b-8317-c90056de43ba","task_key":"default--validation--7891","task_revision_id":"1","upstream_id":"7891","short_description":"HellaSwag validation 7891","config":"default","split":"validation","body":"{\"activity_label\":\"Baking cookies\",\"ctx\":\"There's a tutorial on how to make shortbread cookies. the chef\",\"endings\":\"['looks at a flyer on a screen.', 'demonstrates how to mix the butter and flour in a blender to blend the dough.', 'grabs the bread and slices it up and out then adding the rest of the bread.', 'is giving directions as he uses the knife and the cheese, and shows how you can spread the milk, and how it can be baked before adding instead the baking powder.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}