{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"98212586-96e1-57af-a110-d32a85d11aba","task_key":"default--validation--10890","task_revision_id":"1","upstream_id":"10890","short_description":"HellaSwag validation 10890","config":"default","split":"validation","body":"{\"activity_label\":\"Playing harmonica\",\"ctx\":\"A bald man sits in a wooden chair and talks to the camera while holding, pointing to and occasionally playing a harmonica in his hands. the camera\",\"endings\":\"['pans back to the man doctor sitting in a chair and further inputs appear.', 'man pans the harmonica all around the room while he continues to rock back and fourth to music.', 'zooms in for a tighter shot of the man as he talks and before the man pulls the harmonica directly up to his face and plays the harmonica briefly before pulling the harmonica away and looking into the camera.', 'pans right to show several people in the background who work in the shop.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}