{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"a4e25b68-3356-5737-90a5-8f80da584b42","task_key":"default--validation--8926","task_revision_id":"1","upstream_id":"8926","short_description":"HellaSwag validation 8926","config":"default","split":"validation","body":"{\"activity_label\":\"Croquet\",\"ctx\":\"The boy shows off the objects used to play crochet and then begins hitting the ball around a yard. the boy\",\"endings\":\"['calls for a dog to ask him to fetch some treats and then sits by his toys.', 'then flips it the crochet and then plays again, this time moving it to his ankles instead of his hands.', 'runs around while playing, dropping sticks.', 'continues speaking to the camera while holding the stick and continuing to demonstrate.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}