{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"f0609efe-c5f7-580d-b7e4-17bc4a090cad","task_key":"default--validation--9865","task_revision_id":"1","upstream_id":"9865","short_description":"HellaSwag validation 9865","config":"default","split":"validation","body":"{\"activity_label\":\"Washing face\",\"ctx\":\"When the rabbit stops licking its right side it briefly faces forward and its left back foot scratches its left ear then he licks his left paw. the rabbit\",\"endings\":\"['stops licking his left paw and finally falls into a pit of its own food on the ground.', 'remains where it is and has a large bite of its own.', 'stops licking its right ear and just stares at the camera while the man is now licking its front paw.', 'then lifts its right paw and scratches his right ear and then licks that paw.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}