{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"d2060144-770d-52dc-bf1d-2afe2ed1fa85","task_key":"default--validation--45867","task_revision_id":"1","upstream_id":"45867","short_description":"HellaSwag validation 45867","config":"default","split":"validation","body":"{\"activity_label\":\"Sharpening knives\",\"ctx\":\"He is now outside of a car and is demonstrating how to use the frosted edge of a car window to sharpen the knife. He is back in the garage and explains and demonstrates that using a knife to sharpen another knife. he\",\"endings\":\"['is then shown sharpening a attachment to the phone by pressure.', 'then shows how to shirt the knife with a tissue to get the cleaned and sharpened metal against the clear surface.', 'then asks for people to like the video and to subscribe and thanks everyone from watching.', 'then demonstrates how to snow melt the ice into the car window, then use a brand new folding knife not bothering to put any excess coated tissue paper in it to get the wet.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}