{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"386a8ab2-08b2-558e-8de2-7c2ef1191464","task_key":"default--validation--38909","task_revision_id":"1","upstream_id":"38909","short_description":"HellaSwag validation 38909","config":"default","split":"validation","body":"{\"activity_label\":\"Having an ice cream\",\"ctx\":\"Ice cream gets on her nose as she's licking the cone. As she continues to eat her ice cream ,she gets some on her arm and licks it off. while\",\"endings\":\"['the woman is eating, a group of people are seen going up a snowy hill surrounded by snowslides.', 'she stands up trying to see what is going on, someone lays the ice cream on her face.', 'she continues to eat, the boy next to her begins to sing while being dragged behind him.', 'looking around ice cream drips down her chin.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}