{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"6468b860-b482-5d43-a071-021eb0bc1a3b","task_key":"default--validation--27373","task_revision_id":"1","upstream_id":"27373","short_description":"HellaSwag validation 27373","config":"default","split":"validation","body":"{\"activity_label\":\"Sharpening knives\",\"ctx\":\"The man then grabs the knife and starts gliding it over the block very gently and smoothly before hitting it over a silver steel stick. when finish\",\"endings\":\"['and the knife is seated back into the black case, the man takes it out again and writes on the back of the case along with the words \\\" pizza! vasquez cooking \\\" and from the words \\\" gemma\\\\'s recipe \\\" he takes a look at the ingredients and then writes and separates them neatly.', ', the object is removed from the back of the refrigerator and shown to be a neatly cleaned bbq fry pan.', ', he grabs a piece of paper and shows how it cuts so effortlessly.', \\\"its only the knife's edge.\\\"]\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}