{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"1116de7e-4a4d-5d54-82d5-508ad26d9580","task_key":"default--validation--35890","task_revision_id":"1","upstream_id":"35890","short_description":"HellaSwag validation 35890","config":"default","split":"validation","body":"{\"activity_label\":\"Disc dog\",\"ctx\":\"The dog is black and white and the lady is throwing an orange frisbee around to her to catch in her mouth. each time the girl throws it\",\"endings\":\"[', another dog runs by as the lady continues to find her treats.', ', the dog catches it and returns it anticipating more.', ', the dog with its face caught on the frisbee and the lady gets a blue drink from the fridge at the end.', 'the dog chases it and runs after it and while the woman finally catches the brown frisbee and throws it as far as she can.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}