{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"d7c97375-9f07-5b29-b667-865979776ff9","task_key":"default--validation--26371","task_revision_id":"1","upstream_id":"26371","short_description":"HellaSwag validation 26371","config":"default","split":"validation","body":"{\"activity_label\":\"Walking the dog\",\"ctx\":\"Another car is parked and the cat comes to a halt again before perusing the rest of the sidewalk and walking over the puddles. finally\",\"endings\":\"[', it scoops up the ice off the sidewalk and returns to putting in more everywhere until it reaches the snow.', ', the cats grab their bags and leave.', ', the walk ends and the cat begins walking up the steps and looking at the person who was walking him.', 'they arrive at an empty table where a man is talking and the cat walks findings first on a dog and dog hin.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}