{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"eb3ac337-381d-541b-a60b-103a08d7b9f9","task_key":"default--validation--46142","task_revision_id":"1","upstream_id":"46142","short_description":"HellaSwag validation 46142","config":"default","split":"validation","body":"{\"activity_label\":\"Drinking coffee\",\"ctx\":\"A squirrel is standing on all fours at the edge of a couch. The squirrel leans into a coffee cup using its front limbs. the squirrel\",\"endings\":\"['pushes a rubix cube down, gets up, and begins picking up pieces using its front legs.', 'bobs his head in and out of the coffee cup.', 'brushes an egg which hangs off the coffee cup and onto the glass coffee cup.', 'then eats all the plastic wrap that is on the coffee while keeping one eye on the phone for a while.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}