{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"6101139f-78d1-5390-b18e-b248154c7052","task_key":"default--validation--17507","task_revision_id":"1","upstream_id":"17507","short_description":"HellaSwag validation 17507","config":"default","split":"validation","body":"{\"activity_label\":\"Playing drums\",\"ctx\":\"The boy begins playing the drums. The boy continues playing the drums as smoke effects begin to appear. the boy\",\"endings\":\"['then drops the drumsticks, poses, then walks off of the stage.', 'stops playing and finds himself on a long floor.', 'tries hard to keep the smoke inside but ends up coughing.', 'continues to play the drums with the rest of his hands as smoke begins to enter on his face.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}