{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"f5d12731-f9ed-50b4-80cb-159f8139d6e9","task_key":"default--validation--21241","task_revision_id":"1","upstream_id":"21241","short_description":"HellaSwag validation 21241","config":"default","split":"validation","body":"{\"activity_label\":\"Making a cake\",\"ctx\":\"A persons hands are seen laying clay out onto a cake and begins painting objects on the side. The woman puts the decorations around the cake and continues decorating more objects on the side. she\",\"endings\":\"['eventually ends by painting the cake a series of different colors with many different ending lights attached.', 'finishes the cake by adding the final decorations and presenting it to the camera.', 'moves all the stem away from the cake and continues laying wax.', 'takes a long piece of paper off and tries to make holes through the edges and ends with her putting the piece of paper down.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}