{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"294b8ca8-6b3b-54df-9e04-936ef1fd0deb","task_key":"default--validation--40490","task_revision_id":"1","upstream_id":"40490","short_description":"HellaSwag validation 40490","config":"default","split":"validation","body":"{\"activity_label\":\"Scuba diving\",\"ctx\":\"A person underwater in snorkeling gear is now swimming around the bottom of the water and meets up with another snorkeler and they both go into a ship that is underwater as they both hold their flashlights and cameras to explore the ship and everything else at the bottom of the water. A school of fishes appear and they're swimming and hanging around the boat. the focus\",\"endings\":\"['goes back to the snorkelers as they continue to explore the area with their flashlights.', 'then returns to people underwater with their watches up as well as their mouths open and swim around the ocean with eyes open and cat like looks.', 'then shifts and we see close up of a mirror that basically appears to be a fish.', 'returns to the water from the camera and at first the man is underwater looking beautiful and smiling to the camera and after awhile he notices the camera man is doing the same thing like a dog on a station.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}