{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"4245fb78-4438-55a4-bc33-173c147a9e67","task_key":"default--validation--39912","task_revision_id":"1","upstream_id":"39912","short_description":"HellaSwag validation 39912","config":"default","split":"validation","body":"{\"activity_label\":\"Clipping cat claws\",\"ctx\":\"The girl holds the cats paw in her left hand. The cat groomer grabs a cat nail clipper in her right hand. the cat groomer\",\"endings\":\"['squeezes the cats paw and proceeds to clip the cats nails.', 'shines the nail of the girl.', 'stops to give the cat paw back to the woman.', \\\"takes the cat's left paw and with her open hand cut the nails with latex less nail clippers.\\\"]\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}