{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"851c013c-2336-534e-9a17-2584855fc34b","task_key":"default--validation--2027","task_revision_id":"1","upstream_id":"2027","short_description":"HellaSwag validation 2027","config":"default","split":"validation","body":"{\"activity_label\":\"Computers and Electronics\",\"ctx\":\"[header] How to use duolingo [title] Sign up via facebook or email. [step] When you visit duolingo's main page, you'll be asked to sign up in one of two ways. [substeps] If you choose to sign up via email, you can link your facebook account to duolingo later.\",\"endings\":\"[\\\"If you select to sign up via email, you will receive an email verifying your availability for duolingo. You can also sign up on duolingo's website with your facebook or email address and password.\\\", \\\"If you choose to sign up via facebook, it'll be easier for you to invite friends later. You can also automatically share updates and progress on your facebook timeline.\\\", 'Even if your email address is already on duolingo\\\\'s timeline, you will be prompted to enter the email address you\\\\'ve chosen. You will then be asked to type in your password and tap the \\\" sign up \\\" button.', \\\"Otherwise, you'll be directed to the official form at. You can ask either computer admin or the catalog administrator for the application and its features.\\\"]\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}