{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"4eb1939d-8720-5b0c-83e5-449e8a8734a5","task_key":"default--validation--39320","task_revision_id":"1","upstream_id":"39320","short_description":"HellaSwag validation 39320","config":"default","split":"validation","body":"{\"activity_label\":\"Cleaning windows\",\"ctx\":\"You get a wash cloth and you put it under the faucet to get wet and then you rinse it out so it's not soaking. Then you use it to wash the mirror wrong back and forth and all around. with a dry cloth\",\"endings\":\"[\\\"you put it under the faucet so you don't have to wash your hands.\\\", \\\"it's dry, it's dry, it's clean, it's shiny and it looks like you brushed it.\\\", 'you go over it and then your mirrors are clean.', ', you and cold water with soap and you are done.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}