{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"4ca9d291-e012-50e8-b284-3d5ab7ed4314","task_key":"default--validation--40488","task_revision_id":"1","upstream_id":"40488","short_description":"HellaSwag validation 40488","config":"default","split":"validation","body":"{\"activity_label\":\"Scuba diving\",\"ctx\":\"A view from the water begins and a few houses and a lot of trees are shown on land. a person underwater in snorkeling gear\",\"endings\":\"['is now swimming around the bottom of the water and meets up with another snorkeler and they both go into a ship that is underwater as they both hold their flashlights and cameras to explore the ship and everything else at the bottom of the water.', 'goes to ocean floor swimming air.', 'appears on screen, busy with the snorkeling game and then shows up with a sandwich in his hand.', 'shoes and goggles examines the shorefront of a big wave where some swimmers are as the land continues to rise up in the water.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}