{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"75d8d11a-70ac-5040-9833-7144b034cf40","task_key":"default--validation--29819","task_revision_id":"1","upstream_id":"29819","short_description":"HellaSwag validation 29819","config":"default","split":"validation","body":"{\"activity_label\":\"Sharpening knives\",\"ctx\":\"The man then removes the object and starts rubbing the knife along the hand towel that was under the object and he picks up another tool and begins rubbing the knife along it and once again rubs the knife down with the hand towel. the man\",\"endings\":\"['then wipes the object quickly on a paper towel and begins to use it in several directions before stopping for a second.', 'stops rubbing and picks up another object that he now uses as a mini dustpan and a knife to scrape the excess off the object.', 'then grabs a piece of paper, and while talking he starts to cut it while holding it in the air.', 'ends by scraping the knife along the same table surface surface again.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}