{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"879a3d92-3a03-57a7-8cd1-71860f877c57","task_key":"default--validation--8941","task_revision_id":"1","upstream_id":"8941","short_description":"HellaSwag validation 8941","config":"default","split":"validation","body":"{\"activity_label\":\"Roof shingle removal\",\"ctx\":\"The camera moves around so you can see all the different angles, it looks like the worker is on break he even has a drink like maybe he just had lunch. some new shingles\",\"endings\":\"['are being worked on, the lot is covered in shingles fully inflated so you can see the design works with just the cutter.', 'are being put in to replace whatever was there before.', 'are falling and he starts to fix it up he does a little bit huffing until it is done.', \\\"are now falling off the roof and they're in a big pile like logs.\\\"]\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}