{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"177e4b97-79f9-5550-bc01-a435564fb50f","task_key":"default--validation--14948","task_revision_id":"1","upstream_id":"14948","short_description":"HellaSwag validation 14948","config":"default","split":"validation","body":"{\"activity_label\":\"Gargling mouthwash\",\"ctx\":\"A woman is seen walking into frame and nodding at the camera that leads into her pouring mouthwash out and into her mouth. she\",\"endings\":\"['rinses the mouthwash around, gurgles the liquid into her mouth, then spits it out and smiles.', 'spits out the mouthwash in the end and spits it back out by stepping into frame again.', 'then spits it out into a cup and spits it back out in the end ending with her looking sad and smiling to the camera.', 'spits out more mouthwash and mixes it all together while speaking to the camera.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}