{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"d09e09f9-79f2-5443-82fe-1a083781cd75","task_key":"default--validation--34419","task_revision_id":"1","upstream_id":"34419","short_description":"HellaSwag validation 34419","config":"default","split":"validation","body":"{\"activity_label\":\"Installing carpet\",\"ctx\":\"The group of men in black shirts moved the couch from the wall to other side and removed the other half of the carpet, exposing the green floor. the group of men\",\"endings\":\"['ladder up the wall to the wall and bounce the hem of the brown carpet across the floor.', 'put the carpet on the floor, they measure it, cut it and taped it.', 'continue mopping as four men walked into the room.', 'moved the couch from the wall, revealing the empty hallway.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}