{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"e5a25145-de0c-54b8-9aef-e4121c204d21","task_key":"default--validation--21235","task_revision_id":"1","upstream_id":"21235","short_description":"HellaSwag validation 21235","config":"default","split":"validation","body":"{\"activity_label\":\"Making a cake\",\"ctx\":\"Next she adds 1 teaspoon of vanilla extract and she mixes everything up, then she adds 1 1/4 cups of all purpose flour along with 1/4 cup of unsweetened cocoa powder and salt. Then she adds baking powder and baking soda and she whisks it until smooth. finally\",\"endings\":\"[', the cake was baked consisting of shredded piece of cake and varying amounts of vanilla extract she added to it with the content being browned out to cool.', ', she puts the cake on a plates and begins to roll it all up.', 'she adds a vanilla extract, she puts water in it and she mixes chocolate powder into the candy so it will blend in perfectly.', \\\", she pours the mix into a microwavable safe dish and it's done.\\\"]\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}