{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"17f6bd33-818b-5ffb-a4b7-32273ad4a57c","task_key":"default--validation--9321","task_revision_id":"1","upstream_id":"9321","short_description":"HellaSwag validation 9321","config":"default","split":"validation","body":"{\"activity_label\":\"Sharpening knives\",\"ctx\":\"Various tools are shown laid out while a hand picks up the tools and shows them to the camera. He then grabs another tool and begins sharpening the knife next to some sliced apples. he\",\"endings\":\"['then walks over and picks up one he was slicing.', 'shows off the knife before holding it up.', 'then cuts the apples with the sharpened knife and leads into another person holding and showing off more knives.', 'continues sharpening the knife through the apples.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}