{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"c1931f11-be04-5196-be19-a866da06aa91","task_key":"default--validation--8171","task_revision_id":"1","upstream_id":"8171","short_description":"HellaSwag validation 8171","config":"default","split":"validation","body":"{\"activity_label\":\"Fixing bicycle\",\"ctx\":\"A man is seen speaking to the camera and leads into him holding up various tools and begins putting them on a bike. the man\",\"endings\":\"['then ties the strings on the bike while still speaking furiously to the camera and is seen pushing himself along the side.', 'demonstrates how to properly piece together a bike and peddling the bike to show how it works.', 'tires the tire and then uses a tool to tighten the hole while continuing to speak to the camera.', 'is now riding the bike around the area while speaking to the camera and holding up various obstacles on the bike.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}