{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"fba0f225-bb6a-5d50-8158-e8e5cae1e100","task_key":"default--validation--49424","task_revision_id":"1","upstream_id":"49424","short_description":"HellaSwag validation 49424","config":"default","split":"validation","body":"{\"activity_label\":\"Having an ice cream\",\"ctx\":\"A woman is seen scooping up a spoonful of ice cream and taking a bite with a baby in front of her. the woman\",\"endings\":\"['continues squeezing milk into her spoon and handing the baby back to the woman, after which she bites her ice cream the same way she did the baby.', 'continues to tease the baby with the ice cream giving him little bites here and there as well as taking bites for herself and laughing.', 'then mimics the same bite and finishes by taking the small bite to let go.', 'walks around a city street with shots of shops and bars.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}