{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"9db1cbe7-e482-52f5-88d9-1255e4cd2d04","task_key":"default--validation--26604","task_revision_id":"1","upstream_id":"26604","short_description":"HellaSwag validation 26604","config":"default","split":"validation","body":"{\"activity_label\":\"Having an ice cream\",\"ctx\":\"A man stands behind a food counter and places food in a cup with a stick. He then taunts the customer with the cup, keeping it out of reach. he\",\"endings\":\"['then leisurely starts squeezing lemons into the cup with the stick.', 'then slides it down the counter where she can retrieve it.', 'continues to taunt her by blowing the food into a small circular cup that is sitting on a counter next to the food.', 'does this several times, hitting the guy with it every time.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}