{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"7e67b77e-5abd-5e37-8bb2-2d6ddf76d880","task_key":"default--validation--37911","task_revision_id":"1","upstream_id":"37911","short_description":"HellaSwag validation 37911","config":"default","split":"validation","body":"{\"activity_label\":\"Ice fishing\",\"ctx\":\"The guy reaches into the hole in the ice and pulls on a clear string. The guy brings a fish from the ice which is attached to the string. the guy\",\"endings\":\"['smiles and places the fish on the the ice with both hands.', 'hooks a fish as another person walks to the edge of the hole.', 'gets his fishing pole and puts it in his mouth.', 'lets go of the string and the fish floats to the surface of the ice.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}