{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"221db6b5-c490-5e3c-a5bf-6b855745e72d","task_key":"default--validation--18832","task_revision_id":"1","upstream_id":"18832","short_description":"HellaSwag validation 18832","config":"default","split":"validation","body":"{\"activity_label\":\"Washing face\",\"ctx\":\"A calico cat is seen sitting on a white sheet. The cat closes its eyes as it grooms itself. it\",\"endings\":\"['shaves the pads of its paws.', 'starts licking its paw when the lady stops scrubbing it.', 'uses its tongue to lick its paw over and over.', 'also opens its mouth to speak, showing off its teeth.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}