{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"e2781072-0262-57b6-99b3-ea7f98baf15a","task_key":"default--validation--5120","task_revision_id":"1","upstream_id":"5120","short_description":"HellaSwag validation 5120","config":"default","split":"validation","body":"{\"activity_label\":\"Arm wrestling\",\"ctx\":\"Two women are on a stand opposite one another at a table beginning to arm wrestle. There is a brief hold up before they begin, because their thumb positing is inaccurate. once they\",\"endings\":\"[\\\"have given each other a high five, they hold out the no off sign after each other's initial victory.\\\", 'get to the middle they begin to fold violently and together.', 'have successfully moved their hands, they go back to their team drinking and exhaling while the crowd watches them.', 'finally begin, the girl on the left wins but she is quickly stopped and they have to restart and the same girl wins again.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}