{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"ab0afa8b-ca24-5605-ad61-46985da5035a","task_key":"default--validation--50438","task_revision_id":"1","upstream_id":"50438","short_description":"HellaSwag validation 50438","config":"default","split":"validation","body":"{\"activity_label\":\"Computers and Electronics\",\"ctx\":\"[header] How to prevent others from buying stuff with your amazon echo [title] Open the application. [step] Click on the application in your smartphone or tablet to open it up. Once it is open, look for the \\\" menu \\\" tab in the upper left corner.\",\"endings\":\"['Tap the \\\" menu \\\" tab to open up the options. [title] Tap \\\" settings \\\" in the \\\" menu \\\" tab.', 'Click \\\" accessories \\\" to open up the application. [title] Click the \\\" cost \\\" button beside the card.', 'Under the other three options, click \\\" settings. \\\" from here, select the \\\" server settings \\\" tab to open it.', 'From here you can add and display the contact information. [substeps] You can also choose how you want to share information, such as your username, email address, and password.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}