{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"f00840d9-9119-5a10-8136-d1ab81359490","task_key":"default--validation--31286","task_revision_id":"1","upstream_id":"31286","short_description":"HellaSwag validation 31286","config":"default","split":"validation","body":"{\"activity_label\":\"Having an ice cream\",\"ctx\":\"A woman is sitting with a baby in her arms. She takes licks of an ice cream cone, and the baby laughs hysterically each time. she\",\"endings\":\"['puts the ice cream cone into traced silverware and eats it.', 'playfully kicks the girl across the ice cream.', 'offers him bites, pulling the cone away and making him laugh.', 'presents to the camera several scrunchies she has made as she keeps laughing at the camera.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}