{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"722b17b0-efe9-50be-be94-c2a4890a63a0","task_key":"default--validation--29078","task_revision_id":"1","upstream_id":"29078","short_description":"HellaSwag validation 29078","config":"default","split":"validation","body":"{\"activity_label\":\"Beer pong\",\"ctx\":\"Scenes of two men playing beer pong are shown, interspersed with the two talking to each other exaggeratedly from across the table. One of the men drinks from a cup and falls down unconscious on the floor. the second man\",\"endings\":\"['holds one of the cups and he then shoots arrows at the dead man.', \\\"screams when he's put back down before shooting another cup with a gun.\\\", \\\"kicks the first man, places a ball on the first man's chest, an leaves.\\\", \\\"'s face is shown from several angles as he eventually falls and takes his head down to the table where he is killed.\\\"]\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}