{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"37d7d0f3-e7bd-5e43-a8d8-cfafb9baa3ef","task_key":"default--validation--33903","task_revision_id":"1","upstream_id":"33903","short_description":"HellaSwag validation 33903","config":"default","split":"validation","body":"{\"activity_label\":\"Putting on shoes\",\"ctx\":\"The woman is shown putting her feet into her shoes while they are tied. but\",\"endings\":\"['the laces are removed one at a time, and the woman laces up the other end of the laces and moves herself a few inches to adjust it.', 'after a few minutes of her lifting her feet off of the ground, she takes them off without using her hands.', 'as she puts them on she pretty much falls down on the ground and they put her shoes on her feet.', 'still ends up facing wrong direction.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}