{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"cb55dbe5-8cd3-52f4-a9a7-97ae63edcdf6","task_key":"default--validation--30141","task_revision_id":"1","upstream_id":"30141","short_description":"HellaSwag validation 30141","config":"default","split":"validation","body":"{\"activity_label\":\"Sharpening knives\",\"ctx\":\"A black screen appears and a lot of white words are displayed that include the description of the video and the website. a man and a woman\",\"endings\":\"['are banging on the fifth door of the house while looking out at their beautiful news gear, while the woman talks to him about the work he has done.', \\\"are standing in a kitchen that looks like a business kitchen and he's showing her how to sharpen knives by demonstrating and talking, and then also letting her try to sharpen the knives as well and shows her how to check for sharpness.\\\", \\\"are dancing then it fades to white and they're dancing front to back in rapid getridge style moves and then back to their business as they do various moves each to which the screen then fades to white words that say on the screen that if you see them later on you won't have the time to read them.\\\", 'are seen playing the game of rafting in the water.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}