{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"8dc12535-335c-531f-92b5-16673608a3e3","task_key":"default--validation--19881","task_revision_id":"1","upstream_id":"19881","short_description":"HellaSwag validation 19881","config":"default","split":"validation","body":"{\"activity_label\":\"Gargling mouthwash\",\"ctx\":\"The woman puts a finger under her nose briefly and continues to swish and then goes back to the sink to spit it out, show her teeth, and then rinse out the sink with a bowl. the girl\",\"endings\":\"[\\\"is now facing the camera and another woman is now standing in front of the sink again to continue to hold the cat's nose.\\\", 'walks away from the sink, grabs her mirror and looks at her teeth and continues talking to the camera.', 'then puts toothpaste on her toothbrush then spits out it in the sink, turns on the faucet and starts to wash her mouth with the side of her mouth first.', 'walks past the camera bob, then walks to the camera, smile and stops in front of the sink.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}