{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"c8773ce1-8c89-53b5-9a00-d7a5131f728c","task_key":"default--validation--21408","task_revision_id":"1","upstream_id":"21408","short_description":"HellaSwag validation 21408","config":"default","split":"validation","body":"{\"activity_label\":\"Gargling mouthwash\",\"ctx\":\"They are swishing it in their mouths and almost all of them have their hands over their mouths. one of them\",\"endings\":\"['is screaming at the camera.', 'spits it out in the sink and the other three are trying to carry on.', \\\"has a giant alligator in his hands and he starts pulling on it so that it doesn't bite him as he moves back and forth with the same claw nails.\\\", 'catches one of the girls in his mouth by her face, she hits him and he flashes a big smile at her while she mystery friends are sitting around looking very intrigued.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}