{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"7f15cf16-cca2-57e1-9253-172a894fdb15","task_key":"default--validation--16594","task_revision_id":"1","upstream_id":"16594","short_description":"HellaSwag validation 16594","config":"default","split":"validation","body":"{\"activity_label\":\"Youth\",\"ctx\":\"[header] How to write to a pen pal for the first time [title] Use their name. [step] You don't need to repeat their name a bunch of times throughout the letter, but definitely use their name specifically in the greeting. You might also mention their name one other time later on in the letter.\",\"endings\":\"[\\\"[substeps] Remember that this is not their formal name, but try to be picky about your details, making sure you connect their titles to other names. If you have more than one pen pal, try not to include anyone else's names in the greeting, even if you know they wrote it specifically for your pen pal.\\\", '[substeps] For example, you could say, \\\" hey, i\\\\'m ryanna. How\\\\'s your day going? \\\" [title] Be specific about their name.', '[substeps] You should also mention your own name early on in the letter, even though it may already be on the envelope. This way you complete the introduction and greeting.', '[substeps] Try not to change the name of their pen pal the same way every time you meet them. For example, if you visit their store, avoid using the name \\\" michael \\\" or \\\" parker, \\\" and instead use \\\" emily \\\" instead.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}