{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"2be72469-89a9-50d1-b8de-df2efebfeaf5","task_key":"default--validation--27470","task_revision_id":"1","upstream_id":"27470","short_description":"HellaSwag validation 27470","config":"default","split":"validation","body":"{\"activity_label\":\"Calf roping\",\"ctx\":\"A man is sitting on a horse holding a rope. A calf is released from a pen and he chases after it. the man\",\"endings\":\"['gets off and ties the horses legs together.', 'jumps off his horse and ties the legs of the calf.', 'mounts the horse, and he begins to chase after the calf.', 'ropes the calf again and sets it down while it runs away.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}