{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"e3f8dd7c-327c-57db-8989-c5a0775b33fe","task_key":"default--validation--27524","task_revision_id":"1","upstream_id":"27524","short_description":"HellaSwag validation 27524","config":"default","split":"validation","body":"{\"activity_label\":\"Making a cake\",\"ctx\":\"The list of ingredients are shown on the screen. All three children wipe the baking tray and pour the batter into it. then they\",\"endings\":\"['either use the saran wrap etcetera to move the cookie dough down the tray, or they flip the tray of dough up into the air and cook the cookies on top.', 'stick it in the oven and bake the cake.', 'take a slice of chocolate cake and taste it.', 'spoon it into a bowl with ice, while it is still filled with cookie dough.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}