{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"ff606343-9719-5208-ae94-e1b125c98b95","task_key":"default--validation--5717","task_revision_id":"1","upstream_id":"5717","short_description":"HellaSwag validation 5717","config":"default","split":"validation","body":"{\"activity_label\":\"Cheerleading\",\"ctx\":\"The men flip then the ladies flip and the men lift them in the air and hold them up as the crowd claps. The people forum groups and flip three girls in the air. the people\",\"endings\":\"['stop spinning and poles except for the right and right girl.', 'put the ladies on their shoulders and flip them into the air and spin then do arm wrestling as people run running up.', 'form two groups and hold up two girls who hold up one girl together.', 'flip and the men stand on them.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}