{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"d7fd956f-6b8a-5491-8ed7-a02cc6d593e9","task_key":"default--validation--8942","task_revision_id":"1","upstream_id":"8942","short_description":"HellaSwag validation 8942","config":"default","split":"validation","body":"{\"activity_label\":\"Roof shingle removal\",\"ctx\":\"The camera moves around so you can see all the different angles, it looks like the worker is on break he even has a drink like maybe he just had lunch. Some new shingles are being put in to replace whatever was there before. it\",\"endings\":\"['shows a little distance being taken for a redo and now it shows a young lady putting screws into an old base and putting glue.', 'looks like the job is half way complete and there is only a small portion that needs to be finished.', \\\"'s quite amazing actually they have done it without being damaged beyond repair.\\\", 'appears same exact as the first shingle, it looks different this time but it appears to be a different shape.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}