{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"e7aea028-ff3a-5215-a9da-eeac1628b663","task_key":"default--validation--8347","task_revision_id":"1","upstream_id":"8347","short_description":"HellaSwag validation 8347","config":"default","split":"validation","body":"{\"activity_label\":\"Playing congas\",\"ctx\":\"This man is beating on the congas using the palms of his hands and he also makes fists sometimes to make a deeper sound. he\",\"endings\":\"['gets all pumped up and his shiny shoes are getting shiny as he continues to beat on the congas.', 'also starts stomping on the congas with his feet and the rhythmic beat he made on the congas.', 'is sitting on a black stool and he is in between two microphones to record what is going on.', 'has a start with putting both hands up in the air and he does this for a long time.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}