{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"ece3c7c4-455f-5091-afaf-a596db498c56","task_key":"default--validation--6725","task_revision_id":"1","upstream_id":"6725","short_description":"HellaSwag validation 6725","config":"default","split":"validation","body":"{\"activity_label\":\"Shoveling snow\",\"ctx\":\"A person is standing in the heavy falling snow. They are trying to clear the snow off the sidewalk. the child\",\"endings\":\"['walks towards the car and snow clears it.', 'then gets on their snowboards, puts on their sleds and takes off down the road.', 'just stands with the shovel, not sure what to do.', 'is rubbing the snow off of the sidewalk.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}