{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"23b5776d-0b7b-56e6-868a-c1ce698fd433","task_key":"default--validation--9842--variant-e58fb6b3d3c3","task_revision_id":"1","upstream_id":"9842--variant-e58fb6b3d3c3","short_description":"HellaSwag validation 9842","config":"default","split":"validation","body":"{\"activity_label\":\"Personal Care and Style\",\"ctx\":\"[header] How to keep a secret [title] Know the seriousness of the secret before you hear it. [step] If someone tells you ahead of time that they are going to tell you a secret, ask for more information first. [substeps] Find out if it is a \\\" little \\\" secret or a \\\" big \\\" secret.\",\"endings\":\"['This will tell you how important it is that you keep the secret. It will also tell you if you need to give the person your undivided attention while they disclose their secret (looking at your phone while having a serious talk is poor etiquette).', \\\"It doesn't matter who you tell. Sometimes you can slip up and tell them something you know people will find out.\\\", 'Ask yourself if the person is still forthcoming. Does it really matter what they end up telling you, and how much of it they want to tell you? If so, they may not be telling you the whole truth and you will have to decide how you will find out.', 'If you think it is a small, surprise disclosure, ask them to tell you the truth instead. [title] Know when the right moment for you to talk is.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}