{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"1d0b842a-ab08-5428-a2b0-5d1beb0e35b6","task_key":"default--validation--41823","task_revision_id":"1","upstream_id":"41823","short_description":"HellaSwag validation 41823","config":"default","split":"validation","body":"{\"activity_label\":\"Ice fishing\",\"ctx\":\"The man then picks up his fishing pole and pulls on the fishing wire that's in the fishing hole in front of him, holds the wire up, then puts it back in the hole while he's still talking. the man\",\"endings\":\"['then removes the dink and adjust the fishing line while talking and subjecting to a shot of the hand held camera lens.', 'removes the reel from the fishing hole and begins to fish away while talking to the camera man.', 'then gets off of his bucket and while on his two knees he puts his fingers in and out of the fishing hole and keeps talking.', 'continues talking while showing how to fish, he hangs, pumps his shotgun and pulls with all his might, while the fishing pole catches big fish in slow motion.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}