{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"5f94d07a-34c0-5a82-9511-5f058c913721","task_key":"default--validation--21960","task_revision_id":"1","upstream_id":"21960","short_description":"HellaSwag validation 21960","config":"default","split":"validation","body":"{\"activity_label\":\"Playing drums\",\"ctx\":\"A boy with earphones is drumming and the stick got stuck in the instrument. The band is playing, while the man is singing, the drummer fell off the his chair. the marching band\",\"endings\":\"['unstuck their sticks, and the marching band started their song.', 'started to play and stop.', 'play their drums, then one of the drummers fell on the ground, a person came to help him.', 'is playing important instruments.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}