{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"b4014212-f5ec-506b-8b39-4476bf1fb42f","task_key":"default--validation--41822","task_revision_id":"1","upstream_id":"41822","short_description":"HellaSwag validation 41822","config":"default","split":"validation","body":"{\"activity_label\":\"Ice fishing\",\"ctx\":\"A man with a thick mustache is dressed in winter clothing is sitting on a white bucket turned upside down on a frozen body of water as he drinks a beer from a blue can and talking. the man\",\"endings\":\"['is now cold and he finishes drinking the beer, at which point he gets up from the ice covered bar eating a snow cone.', 'drinks more beers onto a white level alternating between him drinking further and drinking it the rest.', \\\"then picks up his fishing pole and pulls on the fishing wire that's in the fishing hole in front of him, holds the wire up, then puts it back in the hole while he's still talking.\\\", 'then bends over and turns the camera away while the camera continues to move around his body.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}