{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"c4ac221b-9a2e-5a58-b5ff-ae2499185ae9","task_key":"default--validation--11766","task_revision_id":"1","upstream_id":"11766","short_description":"HellaSwag validation 11766","config":"default","split":"validation","body":"{\"activity_label\":\"Starting a campfire\",\"ctx\":\"She then starts cutting a stick and words are displayed at the bottom showing the steps needed to complete the fire. a pocket knife\",\"endings\":\"['is then added to the kindling as she picks out small sticks and lights the fire.', 'is then seen being shown by the young girl and she begins to sharpen the knife again before throwing it to the ground and laying down again.', 'comes into focus and she begins sharpening the knife with difficulty before finally slicing the paper into four pieces.', 'is pulled out and she begins cutting a small piece of tinder.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}