{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"10f3c055-117c-5857-8d71-06c23f312858","task_key":"default--validation--21236","task_revision_id":"1","upstream_id":"21236","short_description":"HellaSwag validation 21236","config":"default","split":"validation","body":"{\"activity_label\":\"Making a cake\",\"ctx\":\"Then she adds baking powder and baking soda and she whisks it until smooth. Finally, she pours the mix into a microwavable safe dish and it's done. she\",\"endings\":\"[\\\"then let's the cake stand for 15 minutes.\\\", 'then mixes all the ingredients and since the safe is full of cookie dough, she sticks them in her oven and bakes them, afterwards gives a thumbs up.', 'then adds some eggs and baking soda.', 'then starts putting the ice cream into a white pot and cooking it with a quick fry pan and an ice cream eating spoon.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}