{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"a97e34a6-34fd-5e6a-bd88-b380e7ad8555","task_key":"default--validation--19879","task_revision_id":"1","upstream_id":"19879","short_description":"HellaSwag validation 19879","config":"default","split":"validation","body":"{\"activity_label\":\"Gargling mouthwash\",\"ctx\":\"An asian woman is talking and she picks up a blue cup and drinks something from it, swishes it in her mouth, lifts up a small listerine bottle and shows it to the camera, lifts up a small heart mirror and continues swishing. the woman\",\"endings\":\"['places her arm in the air and moves it to show off it, grabs the cup and pours that into a glass and pours a little bit into a different glass and shakes it some more.', 'walks on front man, rubs her hand under her nose, and shows her nose to the camera, then drops the small bottle on the table then rubs her nose and licks her fingers.', 'lifts a small heart mirror and pretends to pee under it.', 'then walks off appears to spit in the sink and then gets another cup to drink out of and swishes more liquid in her mouth.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}