{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"2a85e285-f46a-59a5-946b-1a79a0067d80","task_key":"default--validation--27664","task_revision_id":"1","upstream_id":"27664","short_description":"HellaSwag validation 27664","config":"default","split":"validation","body":"{\"activity_label\":\"Sharpening knives\",\"ctx\":\"A man is seen speaking to a woman while showing her how to sharpen a knife. the woman\",\"endings\":\"[\\\"then uses the tool to sharpen the knife with the man's help.\\\", 'then sets her knives down on a rack and the man holds up a tool with the words \\\" and here is how to make chocolate cream \\\" written along.', 'shows off her knife placement and the man continues to speak.', 'is now standing next to him and they continue instructing him in various ways.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}