{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"4ddb6b25-1b50-5b56-bb92-17aa55023c50","task_key":"default--validation--4090","task_revision_id":"1","upstream_id":"4090","short_description":"HellaSwag validation 4090","config":"default","split":"validation","body":"{\"activity_label\":\"Playing violin\",\"ctx\":\"He moves closer to the camera so that you can see the detail of what he is doing with his hands. he\",\"endings\":\"['then removes the camera from his mouth and uses the camera to point at the yard and then proceeds to mop it up.', 'begins to play different notes on the violin as he moves his fingers.', 'then begins to clean the sidewalk with a rag.', 'puts on quite a show attempting to move the grass and swing his body around.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}