{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"1ed35dc6-0eed-542d-a9a2-99872b364bb2","task_key":"default--validation--30012","task_revision_id":"1","upstream_id":"30012","short_description":"HellaSwag validation 30012","config":"default","split":"validation","body":"{\"activity_label\":\"Gargling mouthwash\",\"ctx\":\"A man is indoors, talking and laughing as he holds a small bottle in his right hand. the man\",\"endings\":\"['then sprays down a wooden fence with a liquid.', 'then hits a ball with a stick far above his head several times.', 'brings the bottle closer to his mouth and laughs some more and then pours the liquid from the small bottle onto his mouth and begins to gargle until bubbles come out of his mouth.', 'squeezes the cork with his fingers, releases it into a cup with water, and the man laughs.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}