{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"d7e6451b-e85f-5157-900c-8873ff70201f","task_key":"default--validation--21285","task_revision_id":"1","upstream_id":"21285","short_description":"HellaSwag validation 21285","config":"default","split":"validation","body":"{\"activity_label\":\"Swimming\",\"ctx\":\"A man and a woman are standing upright in a pool, and the man is showing the woman swimming techniques. the woman\",\"endings\":\"['gets out of the pool accepting her swim suit.', 'is sitting and doing calisthenics, and the man is that is showing how to grab onto the her and flip her around.', 'is putting credits into her head, picture of christmas presents buildings, a name of the mother, pictures of her youth and the male swimmer, then the couple appear back in the pool.', \\\"begins to emulate the man's actions, moving her arms and dipping her head under the water.\\\"]\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}