{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"c42fb827-6c03-5896-ab88-f4bb7a96a6f2","task_key":"default--validation--2985","task_revision_id":"1","upstream_id":"2985","short_description":"HellaSwag validation 2985","config":"default","split":"validation","body":"{\"activity_label\":\"Food and Entertaining\",\"ctx\":\"[header] How to make mini vegetable tarts [title] Prepare the puff pastry dough. [step] If the dough was frozen, have it defrost overnight. Roll out the puff pastry dough in a 12 x 8 inch rectangle using a rolling pin.\",\"endings\":\"['Set the dough over parchment paper and cut it into diamond shapes. [title] Preheat the oven to 350ºf/170ºc.', '[title] Preheat the oven to 325 degrees f (177 degrees c). [title] Cream the butter and sugar until light and fluffy.', 'Place it on a baking sheet covered with parchment paper. Leave the dough to chill in the fridge while you prepare the filling.', \\\"Bake it on the stovetop until it turns lightly brown and won't burn. [title] Stand the tarts with a wire rack.\\\"]\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}