{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"07cda022-417e-543d-8e02-238e5ddea7de","task_key":"default--validation--11313","task_revision_id":"1","upstream_id":"11313","short_description":"HellaSwag validation 11313","config":"default","split":"validation","body":"{\"activity_label\":\"Spread mulch\",\"ctx\":\"The man then comes in the back yard with a green wheelbarrow and dumps the dirt in the back yard. He then takes his rake and moves the dirt back and forth around the house evening it out. when\",\"endings\":\"['he is done, they return the two bags to the woman who is now tapping her foot and speaking to the camera.', 'complete, he disappears and a path of mulch is seen along the garage.', 'they are finished, he walks into the house to get done and check out the wood, he puts it on the floor and washes it and plants it in the yard.', \\\"done, it looks pretty cool while it's being put into a trashcan.\\\"]\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}