{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"d8c1ca06-7d08-5579-831b-983a46b32069","task_key":"default--validation--31154","task_revision_id":"1","upstream_id":"31154","short_description":"HellaSwag validation 31154","config":"default","split":"validation","body":"{\"activity_label\":\"Baking cookies\",\"ctx\":\"She mixes a third time and adds chocolate chips, finally mixing one more time. The oven peeps and she prepares a pan with cookie dough. she\",\"endings\":\"['sits down with the cookies in her lap next to the fire.', 'takes a cookie sheet and a can of vanilla extract, put both in a refrigerator, then peels it for 30 and peels it for 50.', 'puts the dough into the oven and presses bake options.', 'scrapes the cake off the oven and shows off the finished product, except for the frosting and that.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}