{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"add452e6-7d29-5879-8645-1f26a9d4d705","task_key":"default--validation--35983","task_revision_id":"1","upstream_id":"35983","short_description":"HellaSwag validation 35983","config":"default","split":"validation","body":"{\"activity_label\":\"Food and Entertaining\",\"ctx\":\"[header] How to throw a surprise birthday party [title] Determine if your honoree will react well to a surprise. [step] Not all people react well to something unexpected. So before planning a surprise party consider how your guest of honor will react.\",\"endings\":\"[\\\"Just like many college-aged people, it's important to gather some pros and cons to determine what a surprise birthday party is for your guest. [substeps] If your honoree has a hard time being surprised, have a surprise surprise party if the guest of honor reacts the best.\\\", 'The more they react, the more pleased they will be. [substeps] Plan a surprise party on a weekend in the summer in august, or on the weekends in september or march.', 'Some may be more mentally and emotionally angry than others. Additionally, some may be curious or critical.', 'If you are unsure, talk to their close friends or family members to feel out the party idea before proceeding. Also consider whether they enjoy parties or like being the center of attention.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}