{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"a37e01c5-bc0d-54d9-bd20-4a3ac52b038d","task_key":"default--validation--26680","task_revision_id":"1","upstream_id":"26680","short_description":"HellaSwag validation 26680","config":"default","split":"validation","body":"{\"activity_label\":\"Beer pong\",\"ctx\":\"The second man magically has two pong balls in his hands then magically adds 10 cups to the table for beer pong. the first guy\",\"endings\":\"['pours beer and the guys play beer pong.', 'suddenly has two cups full of beer and he urrows while all the cups are hanging in the air.', 'tries to raise all the balls and ends up trying to keep in the cup.', 'drops all the balls in the cups before pulling one behind him and dropping them on the floor.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}