{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"607ca754-b4b6-5391-9c44-afa4f989a36c","task_key":"default--validation--46426","task_revision_id":"1","upstream_id":"46426","short_description":"HellaSwag validation 46426","config":"default","split":"validation","body":"{\"activity_label\":\"Scuba diving\",\"ctx\":\"A leaf is floating through the murky ocean water. A group of scuba divers are exploring the bottom of the sea. they\",\"endings\":\"['are stretching out on the leaf shaped diver suspended on the floating leaf thing.', 'are using cameras as they look at the growth and animals.', 'explore look at the coral and other creatures that live on the sea floor.', 'find structurally sound features underwater and work hard to build a raft.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}