{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"0a3aa709-8c91-5c53-961e-63640ed5497a","task_key":"default--validation--5831","task_revision_id":"1","upstream_id":"5831","short_description":"HellaSwag validation 5831","config":"default","split":"validation","body":"{\"activity_label\":\"Fixing bicycle\",\"ctx\":\"A man wearing a beanie is inside a room. He is talking in front of a disassembled bike. he\",\"endings\":\"['shows a small tool that he has installed into the bike.', \\\"cleans the tire and scissored it, replacing areas where he thinks it's damaged.\\\", 'shows the different tools he needs to replace the chain.', 'uses a scraper to remove peeling paint from the tire.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}