{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"1e640bda-d89d-5345-aeae-8dd85ff25a84","task_key":"default--validation--8904","task_revision_id":"1","upstream_id":"8904","short_description":"HellaSwag validation 8904","config":"default","split":"validation","body":"{\"activity_label\":\"Kneeling\",\"ctx\":\"The man hops to his feet and then kneels back down. The man turns to face right and sit son his feet and bounces. the man\",\"endings\":\"['faces the camera and moves his arms as he speaks.', 'then stands up and walks out the door while the boys continue to play ping pong while the referee speaks to the team.', 'does hop scotch while swinging and mainly kneeling.', 'stands and showcases a mini-fridge.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}