{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"02f23dda-2b84-5d37-bd98-cf9b042e07e6","task_key":"default--validation--39770","task_revision_id":"1","upstream_id":"39770","short_description":"HellaSwag validation 39770","config":"default","split":"validation","body":"{\"activity_label\":\"Painting furniture\",\"ctx\":\"The man rub the vaseline with the brush on all sides of the block, then spray paint it with white paint. The man and his dog left the block on top of the table/cabinet. the man\",\"endings\":\"['talk to the dog while he is inside on the leash and he instruct him.', 'went to the shoulder of his car that was parked on the street and the dog followed the man.', 'came back, he is now wearing gray gloves, picked up the block, then wipe the edge of the block with gray cloth.', 'and his dog they nut out the wax, then the dog is shown in the living room.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}