{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"de813f9d-f667-5e68-afb1-bfb6c68671c1","task_key":"default--validation--29457","task_revision_id":"1","upstream_id":"29457","short_description":"HellaSwag validation 29457","config":"default","split":"validation","body":"{\"activity_label\":\"Ice fishing\",\"ctx\":\"He finds a little hole in the ice and sets up to fish, moving around here and there before finally settling in his chair. He starts to move his rod around in the water and pulls up a little fish. he\",\"endings\":\"['finally goes under the water and finishes his fish as the wind picks up and blows the fish off of the boat.', 'enjoys the catch and gets off of the chair to go rub it all over himself.', 'caught himself a 9 inch blue gill, he goes back in and catches another one and he throws it back into the water.', 'sees something up close and turns around and my man is pushing it out of his kayak to fish.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}