{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"5c6d8e96-6869-5892-a999-4c814994da74","task_key":"default--validation--39318","task_revision_id":"1","upstream_id":"39318","short_description":"HellaSwag validation 39318","config":"default","split":"validation","body":"{\"activity_label\":\"Cleaning windows\",\"ctx\":\"How to clean your rv windows and mirrors fast without using any spray. you\",\"endings\":\"['also have a bucket that you spray paint a window in.', 'can reach for a running water hose and clean the inside of your rv quickly.', \\\"get a wash cloth and you put it under the faucet to get wet and then you rinse it out so it's not soaking.\\\", 'meticulously clean the window in the glass shop and then take the plastic off and start taking the hood off.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}