{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"bd686535-bac3-5106-bf82-e55af7a84851","task_key":"default--validation--38406","task_revision_id":"1","upstream_id":"38406","short_description":"HellaSwag validation 38406","config":"default","split":"validation","body":"{\"activity_label\":\"Running a marathon\",\"ctx\":\"Various shots of a person running after dark are shown ranging from the bridge onto a city as well as a long beach. the person\",\"endings\":\"['runs through various buildings as well as a fire and chases him all around.', 'then runs their hands over the houses while running and singing to the camera.', 'keeps running through the city while the camera captures him from several angles.', 'goes down numerous streets and long hills of a neighborhood and fails onto one particular road.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}