{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"86e51431-9bde-5cc9-ac58-d88e40b24baa","task_key":"default--validation--3623","task_revision_id":"1","upstream_id":"3623","short_description":"HellaSwag validation 3623","config":"default","split":"validation","body":"{\"activity_label\":\"Rock climbing\",\"ctx\":\"A white wall is shown with pegs poking out and strips of tape hanging down and a girl begins cliff climbing the wall as a man in a blue shirt begins to talk. the young lady\",\"endings\":\"['climbs along backwards on the rails and we see her jump straight off of the side of the chalk.', 'gets to the bottom and takes her turn but then falls on the ground and ends on her head.', 'continues progressing up the wall as the man continues to talk in a split screen.', 'continues her climb as the man holds up sticks and follows her down each way until she reaches the bottom.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}