{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"0d9f1afe-fb6c-5cbd-817e-c3cb2259a2b4","task_key":"default--validation--18726--variant-7c11d1a966ac","task_revision_id":"1","upstream_id":"18726--variant-7c11d1a966ac","short_description":"HellaSwag validation 18726","config":"default","split":"validation","body":"{\"activity_label\":\"Education and Communications\",\"ctx\":\"[header] How to write fan mail [title] Introduce yourself. [step] Include your name, where you are from, and your age in the letter. Mention your personal connection to the person or the work that they do.\",\"endings\":\"['[substeps] For example, you could write, \\\" hi! I\\\\'m russell lang, your editor! \\\" [title] Talk about the publication. [step] You can send a fan letter on other social media platforms, like the fanbase on youtube.com or pinterest.', 'Provide as many details about your company as possible so that it will be easier for the receiver to determine who you are. [substeps] If you were hiring someone to design a banner, then include a brief account of that job.', \\\"[substeps] You don't need to give your name unless you are both the copyright owners of the work or are a beyonce author. You don't want the reader to know who you are, so you need to keep them guessing about your name.\\\", 'You can also mention anything that you and the person may have in common. [substeps] If you are writing to your favorite basketball player, you might say, \\\" hello, my name is _ _ _.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}