{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"d7469aac-a126-5767-896d-adfa1cd0e390","task_key":"default--validation--13511","task_revision_id":"1","upstream_id":"13511","short_description":"HellaSwag validation 13511","config":"default","split":"validation","body":"{\"activity_label\":\"Sharpening knives\",\"ctx\":\"He shows how to use it and how you can use it with either hand. Then he begins to sharpen one of his knives. after he\",\"endings\":\"['is done sharpening it he demonstrates with a tomato how helpful it is.', \\\"'s done with the knife he puts it down and walks away into the chop shop.\\\", 'is done sharpening, he is shown laying on the floor and smiling.', 'has sharpened enough knives to make them bigger, he slides a smaller knife into a sharp one and starts sharpening it.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}