{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"f0862b20-a961-5975-8b0a-7a80d02f7341","task_key":"default--validation--7390","task_revision_id":"1","upstream_id":"7390","short_description":"HellaSwag validation 7390","config":"default","split":"validation","body":"{\"activity_label\":\"Computers and Electronics\",\"ctx\":\"[header] How to call friends on snapchat [title] Update snapchat. [step] If you haven't updated snapchat in a while, you'll want to grab the latest version so that you have access to the chat 2.0 features, which includes free voice calling. This feature was introduced in version 9.27.0.0, released march 2016.\",\"endings\":\"['There are currently no locked snapchat features enabled on smartphones, with the exception of the 111-23 view feature included in this service. [substeps] These features will be activated by a call center.', '[substeps] Tap the text field in the upper-right corner and select \\\" update \\\" to update your update. If you don\\\\'t already have a live chat, you\\\\'ll need to create a live chat after you open your phone.', \\\"You can update your snapchat app from your device's app store. [substeps] Voice calls are not available in all areas.\\\", \\\"Visit the snapchat page to update. [substeps] This feature allows you to add new friends that you haven't updated during initial updates.\\\"]\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}