{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"6c40297f-4932-5ab2-9250-126b4b894133","task_key":"default--validation--25241","task_revision_id":"1","upstream_id":"25241","short_description":"HellaSwag validation 25241","config":"default","split":"validation","body":"{\"activity_label\":\"Gargling mouthwash\",\"ctx\":\"A boy shows up on the screen and gets a cup of water and gargles it. He smiles and walks off the screen for a moment. he\",\"endings\":\"['returns and continues gragling and laughing.', 'returns with another cup of water and swishes it around until it dries.', 'finally walks back to the table and he dumps the water into the cup and returns to settle into the chair.', 'then appears in a room with a sheet that has chad printed on it.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}