{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"c1696040-f959-5627-9916-151437f32316","task_key":"default--validation--34161","task_revision_id":"1","upstream_id":"34161","short_description":"HellaSwag validation 34161","config":"default","split":"validation","body":"{\"activity_label\":\"Shoveling snow\",\"ctx\":\"The man stretches and puts on his winter clothes. He sprays his shovel with nonstick spray and shovels his driveway. the man\",\"endings\":\"['takes a piece of paper off of his windshield and writes on it.', 'sits in his perfectly cut snow shovel and shovels while showing off the home he has helped create.', 'lays down and had a hard time breathing.', 'brags of his project.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}