{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"5a59206c-5e30-53a1-a214-295b58810aa8","task_key":"default--validation--2765","task_revision_id":"1","upstream_id":"2765","short_description":"HellaSwag validation 2765","config":"default","split":"validation","body":"{\"activity_label\":\"Hanging wallpaper\",\"ctx\":\"She forgets the scissors and walks out to grab them and comes back to apply more paper on the wall. A break is needed and the woman get a drink from the kitchen. shortly\",\"endings\":\"['after, the woman walks back and is shown walking around the garden putting more paper on the wall.', 'after, two women appear playing crochet and the woman introduces that she is the woman who has made the first cut on the wall.', 'after she is back to applying white paint on the wall.', 'after, the lady continues her tasks and then walks outside to take her cat out.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}