{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"noindex","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[]}},"task_id":"d33c9ffb-6780-5c0f-98a3-456ff5f7aba0","task_key":"default--validation--1549","task_revision_id":"2","upstream_id":"1549","short_description":"HellaSwag validation 1549","config":"default","split":"validation","body":"{\"activity_label\":\"Cleaning sink\",\"ctx\":\"A man is seen standing behind a counter speaking to the camera and holding up cleaning products. He puts the products into a sink and begins scrubbing around the sides. he\",\"endings\":\"['wipes down the counter and shows off the finished sink.', 'sprays down all the surfaces and flips over to show the just cleaned sink.', 'moves to one side of the sink and slowly replaces the tile while also speaking to the camera.', 'puts a pants and a shirt into the sink and dries off.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"noindex","subproblems":[],"grids":[]}