{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"a55075eb-72c2-5a2d-baf7-fbccf48de54c","task_key":"default--validation--12223","task_revision_id":"1","upstream_id":"12223","short_description":"HellaSwag validation 12223","config":"default","split":"validation","body":"{\"activity_label\":\"Disc dog\",\"ctx\":\"A man and dog are seen standing outside in a field with a man playing fetch with a dog. The man throws the frisbee around with the dog while the animal brings it back. the man\",\"endings\":\"['has the dog back to play fetch with the dog while the dog keeps chasing after the frisbee.', 'continues to play and dogs fetch around the man.', 'performs several more tricks with the dog using several frisbees.', 'continues to play fetch with the dog and scores a point across the screen.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}